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ABSTRACT 


Modern  Test  and  Evaluation  has  long  supported 
acquisition  of  warfighting  systems  in  the  United  States 
Navy.  As  the  complexity  and  long-term  supportability  of 
these  systems  has  dramatically  increased,  the  need  to 
successfully,  and  incrementally  test  and  evaluate  families 
of  systems,  including  their  interfaces,  has  become  even 
more  critical.  Long  established  techniques  and 
methodologies  for  T&E  may  still  apply,  but  new  factors  must 
be  addressed.  As  the  Navy  continues  to  grapple  with 
acquisition  reform,  and  aims  to  transform  itself  in  the 
future,  the  Warfighters'  needs  have  essentially  remained 
the  same  -  delivery  of  the  best,  most  effective  weapons,  as 
soon  as  possible,  and  made  easy  to  operate  and  maintain. 
Without  an  equally  effective  developmental  and  operational 
test  and  evaluation  process,  the  United  States  Navy  cannot 
satisfy  this  need. 

This  thesis  examines  T&E  today  and  where  it  must  go  in 
the  future.  It  provides  recommendations  for  T&E 
enhancements,  and  explores  several  areas  where  the  Navy, 
and  Joint  Services,  is  already  looking  towards  future, 
integrated  and  collaborative  test  and  evaluation. 
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EXECUTIVE  SUMMARY 


Test  and  Evaluation  (T&E)  is  required  by  law  and 
contract  for  all  major  Department  of  Defense  (DOD) 
acquisition  programs.  The  science  of  T&E  is  currently 
taught  to  a  portion  of  the  Defense  Acquisition  Workforce, 
in  the  career  field  of  T&E.  The  culture  of  T&E  is  embedded 
in  the  corporate  history  of  all  those  who  struggled  to 
defend  the  need  to  both  test  and  evaluate  complex,  critical 
weapons  and  combat  systems  being  developed  and  fielded  by 
the  United  States  Department  of  Defense.  As  the  DOD 
continues  to  transform  itself,  so  must  the  T&E  community 
keep  up  with  the  many  challenges  of  this  transformation, 
including  the  advent  of  evolutionary  acquisition  (spiral 
and  incremental  development)  and  development  in  an  open 
architecture  environment. 

This  paper  strives  to  provide  a  stamp  in  time  of  what 
the  T&E  community  has  been  doing,  what  it  is  currently 
doing,  and  what  can  be  done  in  the  future  to  keep  pace  and 
to  ensure  that  weapon  systems  acquired  on  behalf  of  every 
U.S.  taxpayer  are  tested  and  evaluated  in  a  manner  that 
will  deliver  these  weapons  to  our  warfighters  as  quickly 
and  efficiently  as  possible.  It  is  also  the  responsibility 
of  this  same  community  to  assure  that  the  systems  delivered 
are  the  best  possible  and  will  protect  the  lives  of  those 
on  the  front  lines.  And  given  rapid  deployment  of  these 
weapon  systems,  we  must  ensure  these  systems  perform  as  our 
soldiers,  sailors  and  airmen  need  them  to. 
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I .  INTRODUCTION 


A.  BACKGROUND 

Systems  have  been  tested  in  the  United  States  since 
the  first  weapons  were  developed  for  this  country' s  use  in 
defending  itself,  however  modern  testing  could  be 
associated  with  the  advent  of  nuclear  energy.  The  nuclear 
weapons  age  began  on  July  16,  1945  when  the  U.S.  exploded 
the  first  nuclear  bomb,  codenamed  'Trinity'  at  Alamogordo, 
New  Mexico.  The  "thermonuclear  age"  began  on  November  1, 
1952  when  the  U.S.  exploded  the  first  thermonuclear  bomb  at 
Eniwetok  atoll  in  the  Pacific.  Codenamed  'Mike',  this  bomb 
was  500  times  more  powerful  than  the  'Trinity'  test  and  had 
an  estimated  yield  of  10.4  megatons. 


Figure  1.  Nuclear  bomb  test  Priscilla,  June 

24,  1957 

According  to  the  environmental  lobby  Greenpeace,  the 
U.S.  has  carried  out  1,030  nuclear  weapons  tests  (the  last 
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and  final  test  on  23  September  1993);  the  equivalent  of  one 
nuclear  weapons  test  every  17  days  since  its  first  test 
(Campaign  History,  2002.)  These  test  were  planned  to  be 
successful.  Each  test  was  a  step  in  an  overall  master  test 
plan  that  would  guarantee  success  of  the  program  while 
maintaining  a  broad  enough  region  of  uncertainty  to 
compensate  for  the  unexpected.  These  were  extremely 
regimented  programs.  While  certainly  a  formidable 

challenge  to  appropriately  test  nuclear  weapons,  this  paper 
will  focus  on  conventional  (non-nuclear)  Department  of  Navy 
(DON)  weapon  systems  under  DOD  development. 

For  the  purpose  of  this  research,  and  as  defined  in 
the  original  version  (dated  December  1996)  of  SECNAVINST 
5000. 2B: 

A  "weapon  system"  is  an  overarching  term 
that  applies  to  a  host  platform  (e.g.,  ship, 
aircraft,  missile,  weapon,  combat  system 
subsystem ( s ) ,  component ( s ) ,  equipment ( s ) , 

hardware,  firmware,  software,  or  item(s)  that  may 
collectively  or  individually  be  a  weapon  system 
acquisition  program  (i.e.,  all  programs  other 
than  information  technology  programs) . 

B .  PURPOSE 

The  purpose  of  this  research  is  to  provide  a 
historical  account  of  what  has  been  done  in  the  past,  what 
is  currently  being  accomplished,  and  what  could  be  done  in 
the  future  to  ensure  that  every  weapon  system  acquired  on 
behalf  of  U.S.  taxpayers  is  tested  and  evaluated  in  a 
manner  that  will  deliver  these  weapons  to  our  warfighters 
as  quickly  and  efficiently  as  possible.  And  given  rapid 
deployment  of  these  weapon  systems,  DOD  must  ensure  they 
work  as  our  soldiers,  sailors,  and  airmen  need  them  to. 
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This  research  will  examine  several  factors  that  should 
prompt  an  evolution  in  how  modern  T&E  must  be  conducted. 
T&E  must  continue  to  support  the  many  DOD  weapon  systems 
under  acquisition  at  present,  and  within  the  coming  decade, 
but  it  must  be  agile  enough  to  accommodate  future,  open 
weapon  systems,  which  will  have  potentially  different  sets 
of  requirements  and  risks  to  be  weighed  only  through 
conscientious  and  appropriate  testing  and  evaluation. 

C .  RESEARCH  QUESTIONS 

The  intent  of  this  research  study  is  to  focus  on  a 
variety  of  questions,  some  in  depth,  and  others  less  so,  to 
build  a  case  that  test  and  evaluation  (T&E)  as  it  is 
conducted  today,  must  evolve  to  keep  pace  with  the  DOD  as 
it  undergoes  reform,  transformation,  perpetually  shifting 
requirements,  budget  fluxuations,  and  an  emerging  and 
dangerous  new  set  of  enemies  and  unforeseen  threats.  This 
set  of  questions  can  be  grouped  into  four  themes,  including 
history  and  the  present,  guidance  and  leadership,  open 
systems,  and  T&E  in  the  future. 

History  and  the  Present: 

•  How  are  Navy  weapon  systems  acquired  today? 

•  How  are  US  Navy  surface  combatant  weapon  systems 
evolved  today? 

•  What  is  Test  and  Evaluation,  and  how  is  it  conducted 
in  today' s  Navy? 

Guidance  and  Leadership: 

•  What  is  acquisition  reform,  and  how  does  it  apply  to 
T&E? 

•  What  does  Transformation  mean  with  respect  to  T&E? 

•  What  do  current  Navy  Leaders  think  about  T&E  today? 
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Open  Systems : 

•  What  are  "open  systems"? 

•  What  is  "open  architecture"  and  what  is  the  Navy' s 
commitment  to  OA? 

•  How  will  OA  improve  weapon  systems  in  the  future? 

•  What  are  recent  improvements  to  OA? 

•  What  are  the  advantages  and  disadvantages  of  OA? 

•  What  are  examples  of  existing  OA  systems? 

T&E  in  the  Future : 

•  What  is  required  to  properly  test  and  evaluate  future 
Navy  systems? 

•  What  are  the  recent  changes  in  the  methodology  of 
weapon  systems  computer  program  development? 

•  How  are  Joint  systems  tested  and  evaluated? 

•  What  is  evolutionary  acquisition,  and  how  does  it 
apply  to  T&E  of  those  future  systems? 

•  How  should  T&E  be  taught  to  ensure  future  T&E 
professionals  would  be  prepared  for  future  challenges? 

•  How  must  T&E  evolve  in  the  future? 

It  should  be  noted  that  the  AEGIS  program 
(specifically  the  AEGIS  platform,  the  AEGIS  Weapon  Systems, 
and  the  AEGIS  Weapon  System  Computer  Program)  will  be  used 
extensively  as  a  case  study  when  exploring  many  of  the 
questions  stated  above.  In  addition,  some  attention  will 
be  focused  towards  the  Missile  Defense  Agency,  however 
mainly  as  it  relates  to  the  AEGIS  Ballistic  Missile  Defense 
(ABMD)  development  effort. 
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D.  POTENTIAL  BENEFITS  FROM  THIS  STUDY 

As  a  member  of  the  T&E  acquisition  workforce,  and  a 
T&E  practitioner  for  approximately  the  last  15  years,  the 
author' s  sincere  hope  is  that  there  will  be  several 
benefits  from  this  research  study.  This  research  shall 
provide  recommendations  and  assessments  to  both  DON  and  the 
T&E  professional  acquisition  workforce  on  what  can  be  done 
to  prepare  for  testing  of  future,  open  systems. 

In  addition,  this  research  is  hoped  to  have  actual 
benefit  to  the  Warfighters  of  the  future,  who  will  depend 
on  timely  and  appropriate  testing  and  evaluation,  leading 
to  weapons  on  target,  and  the  ability  to  fight  and  win, 
unhampered  by  systems  which  offer  technology,  but  are  not 
suitably  tested  and  ready  to  go  into  harms  way. 

E.  SCOPE  AND  METHODOLOGY 

1 .  Scope 

The  scope  of  this  research  study  is  divided  into  five 
parts.  The  first  part  is  the  introduction,  and  includes  a 
brief  discussion  on  DOD  acquisition,  acquisition  reform, 
T&E,  and  what  open  systems  means  to  DOD  and  how  the  rush  to 
get  state-of-the-art,  open  systems  to  the  Warfighter, 
presents  a  unique  set  of  challenges  to  both  testers  and 
evaluators . 

The  second  part  involves  a  historical  review  of  test 
and  evaluation,  touching  on  the  acquisition  reform 
discussion  from  the  background  section,  but  going  into  more 
detail.  This  section  will  expand  on  the  history  of  T&E, 
T&E  guidance,  and  T&E  in  practice  today.  This  section  will 
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also  include  a  short  discussion  about  where  T&E  needs  to  go 
in  the  future,  which  is  expanded  in  much  greater  detail  in 
section  four. 

The  third  part  will  focus  on  open  systems  and  open 
architecture,  including  current  guidance  as  related  to  DOD 
systems  under  acquisition  today  and  standards  for  open 
architecture,  applicable  to  weapon  systems  to  be  acquired 
in  the  future.  This  part  will  also  discuss  a  few  ongoing 
examples  of  system  under  current  development,  including  the 
advantages,  as  well  as  challenges,  of  working  with  open 
systems . 

The  fourth  part  will  use  the  findings  from  section 
three,  to  build  a  case  for  T&E  in  the  future. 

Finally,  the  fifth  section  will  present  conclusions 
and  recommendations  for  further  study. 

The  end  result  from  this  research  is  to  contrast  where 
modern  T&E  appears  to  be  headed  in  the  future  and  where  it 
needs  to  go  based  on  the  latest  published  acquisition 
reform  guidance,  and  based  on  where  open  systems 
development  will  effect  future  DOD  development  and  future 
weapon  systems  undergoing  T&E. 

2 .  Methodology 

The  methodology  used  in  this  research  consists  of  the 
following : 

•  Conduct  a  literature  review  of  DOD  and  DON  related 

guidance  and  reports  on  T&E  and  acquisition  reform. 

•  Conduct  an  in-depth  review  of  available  Program 

Executive  Office  (PEO)  level  briefings  and  white 

papers  covering  acquisition  reform,  transformation. 


6 


and  steps  to  address  legacy  systems,  either  in-service 
or  currently  under  development  and  acquisition  today. 

•  Interview  members  of  the  Acquisition  Workforce, 
specifically.  Test  and  Evaluation  Professionals  to 
assess  their  efforts  to  prepare  for  emerging,  open 
systems  to  be  developed. 

•  Interview  members  of  various  Program  Executive 
Offices,  who  are  presently  involved  in  the  acquisition 
of  systems  which  will  be  "open"  from  the  inception  to 
assess  their  opinions  on  how  prepared  we  will  be  to 
test  and  evaluate  their  systems  in  the  future. 

•  Participate  in  T&E  communities  of  practice,  including 
the  International  Test  and  Evaluation  Association 
(ITEA),  and  the  Defense  Test  &  Evaluation  Professional 
Institute  (DTEPI) . 

•  Conduct  in-depth  Internet  research  on  all  topics  to 
determine  what  information  is  in  the  public  domain  and 
to  determine  how  commercially  produced,  open  systems 
are  tested  and  evaluated  today. 

F.  ACQUISITION  TODAY 

Defense  acquisition's  primary  objective  is  to  obtain 
cost-effective,  quality  weapon  systems,  in  a  timely  manner, 
while  meeting  an  operational  need.  Today's  modern 

warfighting  systems  are  acquired  under  a  series  of  DOD 
instructions,  directives  and  regulations.  The  Secretary  of 
the  Navy  implemented  SECNAVINST  5000. 2B  in  December  of  1996 
to  provide  a  framework  for  mandatory  procedures  applicable 
to  all  major  and  minor  DON  acquisition  programs. 
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Acquisition  policy  continues  to  evolve  under  what  has 
commonly  been  referred  to  as  acquisition  reform.  Even  when 
SECNAVINST  5000. 2B  was  written  over  seven  years  aqo,  its 
authors  understood  that  acquisition  would  need  to  evolve 
further.  In  fact,  the  instruction  referenced  a  term  that 
would  become  a  catch  phrase  for  modern  acquisition  - 
"Evolutionary  Acquisition." 

As  stated  in  SECNAVINST  5000. 2B,  "When  an  evolutionary 
acquisition  (EA)  strategy  is  used  to  field  a  core 
capability  and  there  are  subsequent  modifications  to  the 
initial  fielded  core  capability,  such  modifications  shall 
satisfy  a  validated  requirement  and  be  supportable  in  the 
operational  environment.  EA  modifications  to  the  core 
capability  shall  be  funded,  developed,  and  tested  in 
manageable  increments.  Each  increment  shall  be  managed  as  a 
modification . " 

Recently,  the  Secretary  of  Defense,  Donald  Rumsfeld 
called  for  a  new  Department  of  Defense  acquisition  system. 
In  January  of  2001  during  a  nomination  hearing  before  the 
Senate  Armed  Services  Committee,  Secretary  Rumsfeld 
(Canahuate,  2001,  52)  said,  "The  present  weapons  systems 
acquisition  process  is  ill-suited  to  meet  the  demands  posed 
by  an  expansion  of  unconventional  and  asymmetrical  threats 
in  an  era  of  rapid  technological  advances  and  pervasive 
proliferation."  Later  that  same  year,  in  October  of  2002, 
Deputy  Defense  Secretary  Paul  Wolfowitz  cancelled  all 
existing  acquisition  rules,  and  stated  that  new  ones  should 
be  prepared.  At  the  time,  he  provided  interim  guidance 
pointing  to  a  simpler  system  to  "rapidly  deliver 
affordable,  sustainable  capability  to  the  warfighter  that 
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meets  the  warfighter's  needs.  (Caterinicchia,  2003) 
Further,  Secretary  Wolfowitz'  memo,  spoke  of  "transforming" 
the  military,  and  Defense  Secretary  Rumsfeld  has  urged 
civilian  and  military  leaders  to  acquire  new,  high-tech 
systems.  And  once  acquired,  these  systems  must  be  rapidly 
delivered  onto  the  battlefield. 

DOD  acquisition  still  faces  challenges.  In  his  March 
of  2003  resignation  letter  to  President  Bush, 
Undersecretary  of  Defense  for  Acquisition,  Technology  and 
Logistics,  Edward  "Pete"  Aldridge  Jr.  summarized  his  top 
five  goals  for  achieving  "acquisition  excellence"  within 
DOD: 

1)  Improve  the  credibility  and  effectiveness  of  the 
acquisition  and  logistics  support  process. 

2)  Improve  the  morale  and  quality  of  the  acquisition 
workforce . 

3)  Improve  the  health  of  the  defense  industrial 
base . 

4)  Support  the  decision  process  by  rationalizing 
weapon  systems  and  defense  infrastructure  with 
the  new  defense  strategy. 

5)  Initiate  high-leverage  technologies  that  would 
provide  the  war-winning  capabilities  of  the 
future . 

"All  in  all  I  think  we  have  made  significant  progress 
on  accomplishing  these  five  goals  and  setting  in  place  the 
acquisition,  technology  and  logistics  support  activities 
that  you  and  Secretary  [Donald]  Rumsfeld  want  to  have  for 
DOD,"  his  letter  said  (Caterinicchia,  2003,  54.) 
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G. 


EVOLUTIONARY  ACQUISITION  -  EXAMPLE:  MDA 


The  Missile  Defense  Agency  (MDA) , 
released  the  following  information 
acguisition  strategy  (BMD  Fiscal  Year  2004 


in  July  2003, 
regarding  its 
Budget,  2003,  p. 


2.) 


MDA  is  following  an  evolutionary  acquisition 
strategy  for  the  BMD  System  that  effectively 
manages  changes  in  the  threat,  changes  in  BMD 
System  technologies,  and  progress  in  development 
and  testing.  Using  Research,  Development,  Test 
and  Evaluation  (RDT&E)  resources  almost 
exclusively  and  in  conjunction  with  an 
evolutionary  approach,  the  strategy  capitalizes 
on  technological  progression  and  provides  for 
development,  limited  production,  and  deployment 
of  initial  BMD  capabilities  incrementally  as  soon 
as  they  are  ready.  Adopting  an  evolutionary 
acquisition  model,  the  BMD  System  is  constructed 
around  a  "Capability-based  Block"  approach.  Each 
BMDS  Block  spans  a  two-year  timeframe  and 
continuously  builds  capability  into  the  BMD 
System  by  introducing  new  sensor  and  weapon 
projects,  and/or  by  augmenting  and  enhancing 
existing  capabilities.  As  the  new  projects  mature 
they  will  be  integrated  into  the  BMD  System  to 
increase  the  capability  to  respond  to  the 
evolving  threat.  BMDS  Block  management  includes 
decision  points  at  which  activities  will  be 
evaluated  on  the  basis  of  effectiveness  within 
the  overall  system,  technical  risk,  deployment 
schedule,  and  cost.  From  these  decision  points, 
developmental  activities  will  be  accelerated, 
modified,  or  terminated  depending  on  progress  and 
promise . 


H.  TEST  AND  EVALUTAT I ON 


Two  broad  types  of  testing,  which  will  be  discussed  in 
this  document,  are  used  to  assist  DOD  in  meeting  the  goal 
of  defense  acquisition,  as  laid  out  in  section  I.F. 


10 


Developmental  testing  covers  a  wide  range  that  includes 
component  and  systems  engineering  testing,  as  well  as 
modeling  and  simulation.  Developmental  testing  affords  the 
first  chance  to  assess  performance  and  effectiveness  of  a 
weapon  system  against  tolerances  laid  out  in  the  analysis 
of  alternatives.  Operational  testing  will  focus  on 

performance  of  a  fully  integrated  set  of  systems,  ideally 
within  a  realistic  operating  environment.  Testing  at  the 
operational  level  is  the  process  by  which  DOD  assesses 
whether  a  weapon  system  can  satisfy  planned  capability 
before  deciding  to  begin  full-rate  production.  In 

addition,  operational  T&E  uses  independent  assessment  to 
determine  if  a  system  is  effective  and  suitable  for  its 
particular  application. 

DT&E  is  required  for  all  developmental  acguisition 
programs.  For  DON  programs,  the  Design  Agent  (DA)  through 
contractor  testing  or  government  test  and  engineering 
activities  shall  conduct  DT&E.  Combined  developmental 
testing/operational  testing  (DT/OT)  shall  be  pursued 
whenever  possible  to  reduce  program  costs,  improve  program 
schedule,  and  provide  early  visibility  of  performance 
issues . 

The  DOD  Under  Secretary  for  Defense  Acquisition, 
Technology  and  Logistics  (USD  (AT&L) )  /  Defense  Systems 

maintains  a  staff  element  responsible  for  assuring  that 
DT&E  programs  are  sound,  well-executed  and  sufficiently 
address  the  modern  warfighters'  needs.  USD  (AT&L)  refers 
to  this  group  as  Developmental  Test  &  Evaluation.  Their 
mission  (DOT&E  Mission  Statement,  2003,  p.l)  is  to  ensure 
development  of  sound  and  well-executed  test  strategies 
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within  DT&E  programs. 

and 

to  ensure 

that  DT&E  matures  a 

program; 

allowing  it 

a 

good  chance 

of  achieving 

it' s 

critical 

operational 

design  goals. 

This  group 

also 

provides 

the  focal  point 

for  DT&E 

policy  under 

United 

States  Code  Title  10,  Section  133. 


As 

part 

of 

the 

T&E  Best  Practices 

Conference, 

sponsored 

by 

DT&E 

USD 

(AT&L) , 

"T&E  ensures 

our  weapon 

systems 

perform 

as 

desired 

and  meet 

warfighters' 

requirements.  (Weapon  systems  must)  work  when  and  how  they 
are  supposed  to ." (Lockhart ,  Richard,  Integrating  Test  and 
Evaluation,  2002)  The  role  of  T&E  in  the  acquisition 
process  is: 

•  Provide  essential  information  on  which  to  base 
acquisition  decisions. 

•  Assess  technical  performance  and  system  maturity. 

•  Provide  indication  of  program's  development 

progress . 

•  Provide  information  about  risk  and  risk 

mitigation . 

•  Identify  problems  so  they  can  be  resolved  early. 

•  Confirm  weapon  system's  readiness  to  enter  IOT&E. 

•  Advise  on  how  best  to  use  the  system. 

•  Confirm  weapon  system  meets  user  requirements. 

I.  CHALLENGES  TO  THE  DOD 

The  DOD  will  always  face  major  management  challenges 
and  program  risks  as  it  seeks  to  support  and  defend  the 
Constitution  of  the  United  States;  provide  for  the  common 
defense  of  the  nation,  its  citizens,  and  its  allies;  and 
protect  and  advance  U.S.  interests  around  the  world. 
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In  the  latest  report  to  Congress  ("Major  Management 
Challenges  and  Program  Risks,"  2003,  p.  5)  the  GAO 
summarized  these  challenges  into  eight  areas.  Nearly  all 
of  the  major  challenges  to  DOD  apply  directly  to  Test  and 
Evaluation.  From  the  need  to  hire,  train,  sustain  and 
maintain  a  T&E  workforce,  to  having  the  necessary 
infrastructure  (ranges,  targets,  services,  etc.)  in  place 

to  engage  in  meaningful  T&E,  to  having  proper  budgets,  and 
using  technology,  to  keeping  a  mindful  eye  towards  costs 
effectiveness  and  timeliness,  and  monitoring  and  reducing 
risk,  it  seems  T&E's  major  challenges  are  simply  a  subset 
of  what  DOD  needs  to  do  to  improve  and  complete  its 
fundamental  mission.  The  goal  for  the  T&E  community  should 
be  to  help  DOD  along  this  continuous  path  of  improvement. 
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Performance  and 
Accountability  Challenges 


9  Strengthen  strategic  planning  and  budgeting  to  achieve  desired  mission 
outcomes 

•  Hire,  support,  and  retain  military  and  civilian  personnel  with  the  skills  to  meet 
mission  needs 

•  Overcome  support  infrastructure  inefficiencies  to  reduce  costs  and  improve 
operations 

•  Confront  and  transform  pervasive,  decades-old  financial  management 
problems  to  improve  financial  accountability 

•  Effectively  manage  infoimation  technology  investments  to  transform  business 
functions 

•  Improve  DOD's  ability  to  acquire  weapon  systems  in  a  cost-effective  and 
timely  way 

•  Improve  processes  and  controls  to  reduce  oontract  risk 

•  Provide  logistics  support  that  responds  to  the  needs  of  the  warfighter  at  an 
affordable  cost 


Figure  2 . 


DOD  Major  Challenges 


(2003) 
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II.  T&E:  A  HISTORICAL  VIEWPOINT 


A.  LEADERSHIP  PERSPECTIVE 

Steadfast  leadership  by  program  development  managers 
has  brought  us  to  where  we  are  today.  During  the  early 
days  of  nuclear  power,  without  the  strong  and  unfailing 
conviction  of  Admiral  Hyman  Rickover,  who  once  said,  "Good 
ideas  are  not  adopted  automatically.  They  must  be  driven 
into  practice  with  courageous  patience,"  the  nuclear  Navy 
might  never  have  emerged  into  the  uncontested  force  it 
remains  today. 

During  the  early  days  of  Surface  Missile  Ship 
development,  when  existing  weapon  system  were  thought  to  be 
sufficient  to  meet  the  threat,  RADM  Wayne  E.  Meyer 
aggressively  pushed  for  a  fully-integrated  weapon  system. 
His  "build  a  little,  test  a  little,  learn  a  lot,"  approach 
allowed  the  DON  to  field  the  most  capable  surface  combatant 
ever . 


AEGIS  PHILOSOPHY 


BUILD-A-LITTLE 


TEST-A-LITTLE 


LEARN-A-LOT 


Figure  3.  AEGIS  System  Engineering 
Philosophy  (1977) 
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The  Honorable  William  J.  Perry,  Secretary  of  Defense 
from  1994-1997  stated,  "testing  is  the  conscience  of 
acquisition . " 

B.  TEST  AND  EVALUTAT I ON  -  IN  THE  NAVY 


POD  T&E  Organization 


Sec  Army 


ASN  (RD& A) 


H 


Sec  Navy 


DUSA(OR) 


USCINCSOC 

I 


C  of  S, 
Army 


ASA  (ALT), 
PEOs,  & 
PMs 


Commandant, 
Marine  Corps 

i 


 r 

Chief  of 
Naval  Ops 

i) 

Army 
Test  and 
Evaluation 
Command 
(ATEC) 


|  Sec  AF  | _ , 

|  ASAF(AQ) 

|  C  of  S,  AF  | 

Marine  Corps 

Marine 

Systems 

Corps 

Command 

Operational 

(MARCORSYSCOM) 

Test  and 

Evaluation 

Activity 

(MCOTEA) 

Navy 

Operational 

Systems 

Test  and 

Commands 

Evaluation 
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Force 

(NAVAIR) 

(OPTEVFOR) 

(NAVSEA) 

Air  Force 

Air  Force 

Material 

Operational 

Command 

Test  and 

(AFMC) 

Evaluation 

Center 

(AFOTEC) 

NOTE:  Other  Defense  Components  (e.g.  DFAS,  DLA,  DISA,  etc..)  are  also  subject  to  rules  and  regulations  governing  Test  &  Evaluation 


Figure  4.  DOD  T&E  Organization  (2003) 

As  defined  in  SECNAVINST  5000. 2B,  the  following 
guidance  is  provided  with  respect  to  T&E:  "Early 

involvement  between  the  developing  activity  (DA)  and  the 
operational  test  agency  (OTA)  Operational  Test  and 
Evaluation  Force  (OPTEVFOR) /Marine  Corps  Operational  Test 
and  Evaluation  Activity  (MCOTEA)  is  required  to  ensure  that 
both  have  a  common  understanding  of  the  proposed  system 
requirements  and  that  developmental  and  operational  testing 
is  tailored  to  optimize  cost,  schedule,  while  verifying 
performance.  The  Commander,  Marine  Corps  Systems  Command 
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(COMMARCORSYSCOM)  and  Director,  MCOTEA  are  the  principals 
responsible  for  developmental  test  and  evaluation  (DT&E) 
and  operational  test  and  evaluation  (OT&E) ,  respectively, 
within  the  Marine  Corps.  MCOTEA  is  designated  as  the  Marine 
Corps  independent  operational  T&E  activity  responsible  for 
adequate  testing,  objective  evaluation,  and  independent 
reporting  in  support  of  the  Marine  Corps  Acquisition 
Process . 


OPERATIONAL  TEST 
DIRECTOR’S  GUIDE 

Wumander.  operational  test  and 

j!  EVALUATION  FORCE 

NORFOLK  VIRGINIA 


Figure  5.  COTF  OTD' s  Guide  (2001) 

The  Operational  Test  Director' s  Guide  is  an 
instruction  published  and  maintained  by  COMOPTEVFOR  for  the 
benefit  of  OTD's,  OTC' s  and  is  a  valuable  reference  for  the 
entire  DON  T&E  community.  The  OTDG  is  designed  to  provide 
the  operational  test  director  with  guidance  on  the  various 
aspects  of  operational  test  and  evaluation. 


C.  TEST  AND  EVALUATION  -  AEGIS 
1 .  Leading  Up  to  AEGIS 


During  the  World  War  II  era,  aircraft  attacks  against 
naval  ships  became  a  common  threat.  It  became  evident 
during  this  time  that  using  small  (20mm  and  40mm)  and 
medium  (3"  and  5")  caliber  man  crew-served  weapons  against 
enemy  air  threats  was  not  adequate  to  defend  against  the 
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threat.  In  addition,  technological  improvements  in 
aircraft  design  overwhelmed  current-day  Anti-Aircraft 
capabilities  of  US  naval  defense  systems.  Foreign  nations 
were  creating  faster,  more  maneuverable  aircraft,  more 
difficult  to  counter,  and  requiring  greater  manpower. 
Towards  the  end  of  WWII,  a  new  threat  was  introduced  when 
the  first  successful  air-to-surf ace  missiles  became 
available.  These  computer-controlled  missiles  demonstrated 
vulnerability  of  the  surface  ships  due  to  precision 
attacks.  As  a  controlled  experiment  the  Johns  Hopkins 
University  Applied  Physics  Laboratory  (JHU/APL)  started 
developing  surface-to-air  guided  missile  capabilities  on 
TERRIER,  TARTAR,  and  TALOS  (AKA  "3T")  systems  (Lundquist, 
2002,  St  33.)  The  TALOS  Fire  Control  System  was  a  long 
range  Beam-Rider  system  deployed  on  larger  vessels 
including  refurbished  WWII  vintage  Cruisers.  Second  was 
the  TERRIER  system;  a  medium  range  fire  control  system 
deployed  on  smaller  DLG's,  (Large  Destroyer/Light  Cruiser). 
Finally  was  the  TARTAR  fire  control  system,  a  short-range 
missile  system  deployed  on  USS  Adams  DDG-2  class 
destroyers.  These  were  truly  the  first  ships  specifically 
designed  to  handle  a  missile  fire  control  system.  When 
developed,  the  3T's  were  intended  as  direct  replacement  for 
existing  anti-air  gun  system.  The  radar  system  on  these 
ships  reported  range,  bearing,  and  elevation  data,  which 
was  input  to  an  analog  computer  that  determined  the  range 
of  open  fire  and  generated  the  necessary  orders  for 
launching  a  missile  into  the  radar  guidance  beam.  The 
radar  guidance  beam  guided  the  missile  and  developed  the 
necessary  steering  instructions  to  intercept  its  target. 
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This  phenomenal  technology  later  became  the  basis  for  the 
state-of-the-art  SPY  radar  system  on  current  AEGIS  Cruisers 
and  Destroyers. 

The  standing  Chief  of  Naval  Operations,  ADM  Arleigh 
Burke,  had  recognized  the  need  for  surface-to-air  missile 
systems  with  performance  capabilities  greater  than  the 
inherent  capability  of  the  3T.  Projected  advances  in 
threat  speeds  and  the  ability  to  be  subject  to  coordinated 
mass  attacks  would  still  require  an  even  faster  reaction 
time  and  greater  firepower  resulting  in  a  technologically 
advanced  system  to  be  called,  "Typhon" .  Human  reaction 
time  was  no  longer  sufficient  to  defend  against  attacks  of 
larger  magnitude  and  speed,  and  a  computer-driven  system 
was  the  answer  for  faster  reaction  time.  The  Bureau  of 
Naval  Weapons  initiated  Typhon  in  1960.  Most  of  the 
efforts  during  the  development  and  testing  of  the  Typhon 
system  was  to  revolutionize  the  new  radar  system  that  was 
developed  by  the  JHU/APL.  The  Typhon  program  developed 
principles  needed  to  effectively  build  a  more  advanced 
weapons  system.  However,  insufficient  attention  and 

emphasis  were  afforded  to  operational  suitability  and 
support  requirements.  The  state-of-the-art  technology 
available  at  the  time  was  still  too  primitive  to  achieve 
the  performance  goals  sought  within  appropriate  size  and 
weight  requirements.  Lessons  learned  from  the  Typhon 
project  led  to  planning  inception  of  the  AEGIS  program  in 
1963  (Madsen,  1986.) 

By  the  late  1960's,  the  United  States  Navy  realized 
that  reaction  time,  firepower,  and  operational  availability 
in  various  environments  were  not  sufficient  to  counter 
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increasingly  sophisticated  enemy  threats.  The  U.S.  Navy's 
inability  to  adequately  defend  itself  called  for  the 
proposal  of  a  more  advanced  defense  system.  The  Advanced 
Surface  Missile  System  (ASMS)  program  began  in  1965. 
Secretary  of  the  Navy  led  a  comprehensive  engineering 
development  program  for  ASMS.  Following  the  cancellation 
of  the  Typhon  program,  the  ASMS  project  was  launched,  later 
renamed  AEGIS  in  1969.  AEGIS,  which  is  a  term  used  for  the 
armor  of  Zeus  (hence  the  phrase  "under  the  aegis  of"  or 
"under  the  protection  of") .  Integrating  still-evolving 
state-of-the-art  radar  and  missile  systems,  AEGIS  is  a 
complete  system  designed  to  handle  tactical  engagements 
from  detection  through  kill  assessment.  Designed  as  a 
fully  integrated  weapon  system,  it  was  built  to  defend 
against  advanced  air,  surface,  and  subsurface  threats.  The 
AEGIS  computer  program  is  a  set  of  operations  controlling 
and  operating  the  entire  combat  system.  The  AEGIS  Weapon 
System  computer  program  provides  a  fully  automated  response 
to  threats  (via  selection  and  application  of  doctrine 
parameters) ,  normal  operation  of  Command  and  Decision  (CND) 
process,  and  on-line  system  performance  assessments.  There 
are  about  one  million  words  (using  CMS-2Y  language)  in  the 
computer  programs  for  the  AEGIS  computer  system  using  the 
UYK  (General  utilities  Data  processing  Computing) 
processing  unit. 

2.  Test  Methodology 

Prior  to  the  introduction  of  the  first  AEGIS  Cruiser 
the  United  States  Navy  had  already  developed  a  methodology 
for  test  and  evaluation  of  Missile  Fire  Control  Combat 
Systems.  There  were  three  distinct  Missile  Fire  Control 
Systems  deployed  with  fundamental  differences  between  each. 


20 


First  was  the  TALOS  Fire  Control  System;  the  long-range 
beam-riding  system  deployed  on  larger  vessels  including 
refurbished  WWII  vintage  cruisers.  Second  was  the  TERRIER 
system;  a  medium-range  fire  control  system  deployed  on 
smaller  DLG's,  (Large  Destroyer/Light  Cruiser).  The  third 
was  the  TARTAR  fire  control  system;  a  short-range  missile 
system  deployed  on  USS  ADAMS  (DDG-2)  class  destroyers. 
These  were  the  first  ships  specifically  designed  to  handle 
a  missile  fire  control  system  (Lundquist,  2002,  1  33-35.) 


From  the  1960's  through  the  early  1980's,  these  ships 
were  the  front  line  sea  defense  for  the  Navy' s  carrier 
battle  groups.  Most  of  these  systems  directly  supported 
operations  during  the  Vietnam  Conflict  and  TERRIER  and 
TALOS  systems  were  credited  with  kills  of  hostile  aircraft 
during  that  timeframe. 
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However  these  systems  were  designed  with  out-dated 
technology,  requiring  an  extensive  amount  of  maintenance  to 
remain  operational.  Included  in  this  maintenance  was  the 
assembly  of  missile  parts  prior  to  positioning  on  the 
launcher  rail  for  firing.  Computers  were  analog,  (syncros, 
servos,  gears  and  other  discrete  components)  and  required 
constant  adjustment  and  alignment  to  maintain  material 
readiness.  It  was  not  until  the  late  1970'’ s,  when  the 
Navy's  MK-1219  digital  computer,  (the  first  digital  missile 
fire  control  computer)  was  retrofitted  into  existing 
systems.  Insertion  of  technology  was  done  similarly  for 
years  to  come. 

Testing  directly  on  the  platform,  during  major 
upgrades  and  revisions  became  the  foundation  of  the  test 
and  evaluation  process  for  many  years  to  come.  This 
philosophy  was  based  on  emerging  technology  and  an  ever- 
evolving  threat. 

Weapon  systems,  new  and  old,  had  to  be  maintained  and 
tested.  Each  ship  was  evaluated  against  minimum  standards 
to  determine  battle  readiness.  These  "first  generation" 
missile-guidance  ships  were  evaluated  very  much  like 
earlier  ships,  except  that  special  tests  were  included  to 
verify  performance  of  the  fire  control  system. 

Each  ship  was  evaluated  first  on  maintenance.  These 
early  missile  systems  required  a  significant  level  of 
maintenance,  which  kept  the  crew  very  busy.  During 
evaluation  periods,  maintenance  actions  were  randomly 
inspected  along  with  the  maintenance  paper  work  and 
documentation  as  well  as  crew  training  to  perform  the  task. 
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For  ships  undergoing  new  construction,  or  coming 
through  a  major  overhaul  period,  (every  three  years  at  that 
time) ,  a  Refresher  Training,  (REFTRA) ,  and  Combat  Systems 
Ships  Qualification  Test,  (CSSQT)  were  added  to  the  ships 
schedule.  CSSQT  could  be  described  as  the  first  and  only 
end-to-end  combat  system  training.  These  training  and  test 
evolution  periods  had  to  be  completed  before  deployment, 
and  failure  due  to  lack  of  training  or  system  deficiency 
was  a  serious  matter.  Standards  of  evaluation  were  very 
high  for  these  Naval  Combatants.  These  ships  were 

independently  tested  and  integrated  units  that  would  later 
be  required  to  operate  in  a  battle  group. 

3.  At-Sea  Testing 

During  new  construction  or  ship  refurbishment, 
components  of  the  new  weapon  system  are  assembled  and 
integrated  for  the  first  time  when  they  are  installed  on 
the  ship.  Previously,  each  element  had  been  individually 
tested  in  the  factory,  where  each  piece  of  equipment  met 
standards  of  construction  and  performance.  This  was  the 
only  insurance  that  these  components  would  integrate 
properly  within  a  functional  weapon  system. 

Integration  was  carried  out  on  the  waterfront;  a 
process  where  all  the  weapon  system  elements  came  together 
and  were  tested  as  a  system  for  the  first  time.  The 

measurement  of  this  integration  was  conducted  during  a 
daily  maintenance  action  called  a  DSOT,  or  Daily  System 
Operability  Test.  This  test  included  the  generation  of  a 
simulated  target,  the  assignment  of  radar,  the  training  out 
of  the  launcher,  and  the  loading  of  a  test  missile  where 
firing  voltage  was  applied  and  a  tail  cone  light 
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illuminated  if  the  circuit  was  complete.  This  test 
verified  system  performance,  each  day. 

FY  75  CONGRESSIONAL 
CRITERIA  FOR  AEGIS 

"The  conferees  concur  in  the  fact  that  subsequent 

authorization  requests  for  Aegis  will  be  predicated  upon: 

•  Successful  at-sea  testing  that  demonstrates  the  ability 
of  Aegis  to  meet  its  prescribed  performance  objectives... 

•  At-sea  operation  and  maintenance  of  the  Aegis  system 
by  shipboard  personnel... 

•  A  cohesive  integration  plan  specifying  the  interface 
of  Aegis  with  the  platform(s)  and  other  weapon  and 
command/control  systems... 

•  Definition  and  approval  by  both  the  Navy  and  the 
Department  of  Defense  of  the  platform(s)  for  Aegis...” 

CONGRESSIONAL  RECORD  -  HOUSE 
JULY  24.  1974 

Figure  7.  Congressional  Criteria  for  AEGIS 

(FY7  5 ) 

Integration  on  the  waterfront  was  costly,  requiring  the 
numerous  support  engineers  and  shipyard  workers  to  make  it 
all  work.  The  effort  also  took  time,  but  proved  to  be  the 
most  effective  way  to  integrate  weapon  systems  in  those 
days.  Due  to  the  many  unigue  features  of  each  respective 
ship,  waterfront  integration  became  a  "living  process." 
Within  this  process  was  conceived  the  notion  that 
performance  of  a  "Class  Of  Ships"  could  only  be  realized 
based  on  "Individual  Hull"  trend  performance.  This  meant 
that  class  requirements  could  be  applied,  as  milestones, 
for  each  individual  ship  to  satisfy.  However,  compliance 
to  these  standards  differed  vastly  from  ship  to  ship. 
While  some  ships  demonstrated  superior  tracking  radar 


performance,  others  enjoyed  stable  launching  systems.  Mean 
Time  Between  Failure  (MTBF)  was  dependent  on  crew  training 
and  aggressive  diligence  to  maintenance  procedures. 

Across  all  phases  of  integration  and  test,  at-sea 
testing  of  ships  remained  the  best  measure  of  performance. 
Ships  were  not  certified  for  deployment  unless  successfully 
completing  REFTRA  and  CSSQT.  CSSQT  required  live  firing  at 
a  target.  CSSQT,  although  centered  on  a  single  ship, 
involved  the  entire  battle  group  during  missile  firing 
events.  CSSQT  could  therefore  set  the  stage  for 
deployment,  and  provided  insight  into  how  the  various  ships 
might  interoperate  during  tactical  operations.  At  this 
stage,  the  Navy  would  use  land-based  testing  to  decide 
whether  to  purchase  components  for  these  new  missile  weapon 
systems,  but  for  end-to-end  system  certification,  at-sea 
tests  was  required. 

4.  AEGIS  Weapon  System  Development 

A  principle  design  goal  of  the  AEGIS  Weapon  System  was 
to  apply  technology  in  such  a  way  to  build  a  new  system  far 
superior  to  that  of  currently  fielded  missile  fire  control 
systems.  Weapon  system  complexity  was  a  main  challenge. 
In  the  earlier  systems,  computers  had  transitioned  from 
analog  to  digital.  The  1219  computer  was  limited  to  64K 
Bytes.  Computer  programs  for  AEGIS  were  envisioned  at 
millions  of  Bytes  and  the  computer  was  completely 
different.  The  first  AEGIS  computer  used  was  the  AN/UYK-7 
in  a  4-bay  configuration,  bringing  computing  power  never 
before  realized  to  the  missile  fire  control  system.  To 
properly  test  this  new  AEGIS  system,  new  methods  of  ET&E, 
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from  the  element  level,  to  the  system  level,  would  have  to 
be  pioneered  -  a  lofty  challenge  that  still,  to  this  day, 
is  evolving. 

AEGIS  integration  and  test  was  carried  out  at  a  number 
levels,  closely  monitored  by  the  Navy's  technical 
factory/manufacturer  representative  (TECHREP)  at  the 
various  manufacturers  that  furnished  AEGIS  equipment 
components  or  assembled  and  tested  AEGIS  components. 


AEGIS  WEAPON  SYSTEM  PRODUCTION 


Figure  8.  AEGIS  Development  &  Testing  (1983) 


a)  COMPONENTS  -  As  specific  pieces  of  the  weapon 
system  are  being  built,  parts  are  constructed 
to  DOD  standards  for  manufacturing  and 
reliability.  Each  component  is  tested  once 
installed  in  the  element  piece. 

b)  ELEMENT  PIECES  -  Each  circuit  card  was  tested 
before  installation  into  its  respective 
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cabinet.  Cabinets,  when  assembled,  were  tested 
individually  to  weapon  system  specifications. 

c)  WEAPON  SYSTEM  COMPONENTS  -  Each  weapon  system 
component  was  assembled  in  a  production  test 
center  (PTC)  where  integration  and  testing 
would  be  conducted  at  a  "System  Level" 
configuration.  This  testing  was  the  basis  for  a 
level  of  performance  that  had  to  be  duplicated 
at  the  shipyard  for  waterfront  integration. 

d)  COMPUTER  PROGRAM  -  Unlike  previous  missile  fire 
control  systems,  computer  program  testing  for 
AEGIS  had  to  be  started  in  a  land-based 
environment,  with  interfaces  being  simulated. 

After  initial  land-based  testing  was  completed, 
where  the  program  was  checked  out  in  tactical 
hardware  resident  at  the  land-based  test  site, 
the  program  was  then  delivered  to  the  PTC, 
where  it  was  installed  into  the  actual  tactical 
equipment  that  would  be  delivered  to  a  specific 
ship . 

5.  Land-Based  Testing 

Computer  program  integration  has  evolved  with  the 
technology.  The  difficulty  in  performance  verification  is 
compounded  when  these  dramatically  more  complex  systems 
(that  these  programs  are  designed  to  control  have)  vary  in 
configuration  from  ship  to  ship.  Upgraded  components  and 
configuration  corrections  in  support  of  an  ever-evolving 
system,  although  not  by  design,  ensured  that  each  ship 
would  be  unigue.  Even  though  the  "ship  class"  held  to 
standards  of  performance,  each  ship  would  find  subtle,  and 
sometimes  not  so  subtle,  variations  that  would  need  to  be 
addressed  through  crew  training  and  proficiency. 

Land-based  testing  of  the  AEGIS  computer  program  is 
currently  conducted  at  several  locations.  The  primary  site 
for  this  testing  was,  and  remains,  the  Combat  System 
Engineering  Development  Site,  (CSEDS) ,  in  Moorestown  New 
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Jersey.  This  site  was  designed  to  house  sufficient  "end 
item"  weapon  system  equipment  to  provide  both  system  test 
and  operator/crew  training. 


Figure  9.  AEGIS  Combat  System  LBTS 
Development  (1977) 

Another  facility  is  the  Production  Test  Center, 
located  in  Moorestown,  New  Jersey,  where  all  of  the 
individual  components  of  the  ships  weapon  system  are 
assembled  and  tested,  and  computer  program  development  and 
testing  is  executed.  In  the  early  stages  of  construction 
and  "sell-off",  confidence  in  the  capabilities  of  the 
system  is  traditionally  high.  The  methodology  for 
development  and  deployment  of  the  first  AEGIS  system  was  to 
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"build  a  little,  test  a  little,"  a  paradigm  that  has 
remained  consistent  into  current-day  testing. 

6.  AEGIS  At-Sea  Testing 

Proven  through  history,  land-based  testing  is  not,  by 
itself,  sufficient  to  certify  performance.  At-sea  live 
fire  testing  is  required.  During  development  of  the  AEGIS 
Weapon  System,  at-sea  testing  was  required  before  the 
system  was  released  for  production.  After  extensive  Land- 
Based  testing  at  CSEDS,  a  pre-production  system  was  sent  to 
the  USS  NORTON  SOUND,  a  converted  WWII  seaplane  tender,  and 
became  the  home  of  the  first  AEGIS  Weapon  System.  The 
system  was  installed  and  a  massive  test  program  was 
initiated.  At-sea  operations  were  conducted  in  stand-alone 
modes  and  also  with  other  naval  units  when  opportunities 
allowed.  Multiple  live  fire  events  were  conducted  and 
AEGIS  eventually  proved  to  be  a  capable  and  flexible 
replacement  for  the  already  aged  TARTAR,  TERRIER,  and  TALOS 
systems . 

After  the  release  for  production  was  given,  the  next 
phases  of  At-sea  testing  were  completed  during  the  new 
construction  period  at  the  shipyard.  Prior  to  AEGIS, 
shipyard  integration  did  not  include  a  live  missile-firing 
event.  To  this  day,  each  AEGIS  combatant  is  required  to 
fire  at  least  one  missile  during  shipbuilder' s  trials  prior 
to  custody  transfer  of  the  ship  to  the  Navy.  Even  in 
today's  budget-conscious  environment,  builder's  trials 
missile  firings  are  mandatory  for  compliance  to  integration 
requirements . 

The  T&E  community  is  continually  challenged  to 
demonstrate  regression  performance  has  been  assured  from 
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ship  to  ship,  and  baseline  to  baseline.  Each  ship,  once 
constructed,  must  pass  the  same  CSSQT  requirements  that 
previous  systems  have  been  required  to  satisfy.  As  the 
complexity  of  each  follow-on  AEGIS  system  has  grown,  so 
must  the  level  of  testing  that  is  completed  during  each 
CSSQT. 

AEGIS  CSSQTs  and  live  firing  events  have  specific  test 


objectives,  with 

respective  measures  of 

performance 

that 

simply  cannot 

be 

tested 

in  a  land-based 

environment 

and 

which  often 

requires 

live  ordnance 

to  satisfy 

the 

ob j  ective . 

To 

ensure 

a  high  standard 

of  testing 

is 

maintained,  AEGIS  test  objectives  are  developed,  approved, 
certified,  and  evaluated  by  the  entire  AEGIS  technical 
community . 

Over  the  last  23  years,  thousands  of  AEGIS  live  fire 
test  events  have  been  completed  at  sea.  The  AEGIS 
community  maintains  a  controlled  closed  loop  engineering 
process  that  monitors  system  improvements  and  makes  sure 
that  ET&E  events  are  at  a  level  sufficient  to  adequately 
test  the  performance  of  the  system.  Whereas  every  AEGIS 
ship  commissioned  by  the  Navy  is  quantifiably  unique, 
testing  of  specific  measures  of  performance  is  required  on 
each  and  every  ship  of  the  class. 

7 .  Aegis  in  the  Future 

As  the  AEGIS  Weapon  System  evolved,  the  ship  classes, 
which  carried  the  TARTAR,  TERRIER,  and  TALOS  systems,  were 
decommissioned.  These  early  missile  systems  led  to  the 
development  of  newer  systems,  including  AEGIS,  and  pushed 
the  limits  of  what  T&E  could  provide.  The  latest  versions 
of  the  AEGIS  Weapon  System  continue  to  evolve,  and  so  must 
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the  state  of  testing  and  evaluation.  As  today's  threat 
evolves,  so  must  future  weapon  systems.  Testing,  whether 
land-based,  at-sea  or  via  modeling  and  simulation,  must 
continue  until  the  last  ship  slips  away  and  a  newer  ship 
takes  on  the  roll  of  defender  of  the  fleet. 

D.  TEST  AND  E VALUTAT I ON  -  LOOKING  FORWARD 

The  future  of  T&E  should  trace  to  the  requirements  and 
features  of  current  and  evolving  threats,  and  in  the 
designs  and  advanced  concepts  for  future  weapons  and  combat 
systems.  For  T&E  to  continue  to  provide  the  confidence  and 
assurance  in  feasible,  effective  and  suitable  future 
systems,  it  must  become  more  agile  and  more  embedded  in  the 
process  of  acquisition.  T&E  consists  of  major  and  minor 
milestones  across  the  acquisition  timeline,  which  take  time 
out,  or  away,  from  the  program  development.  It  is  at  these 
times  that,  ideally,  the  design  must  freeze,  and  in 
essence,  a  snapshot  in  time  is  of  80  taken  in  terms  of 
performance  and  adequacy  of  design.  Did  we  meet  our 
specifications?  Did  we  achieve  expected  tolerances?  Did 
we  get  it  right,  in  terms  of  where  we  are  at  along  the 
development  cycle?  But  in  the  future  state  percent 
solutions,  and  considering  dramatically  shrinking 
timelines,  future  T&E  must  be  ingrained  into  the  fabric  of 
design  and  development. 

Further,  the  premise  of  operational  testing,  including 
evolving  operator  needs,  must  be  considered.  The  impact 
from  the  realization  that  suitability  and  effectiveness  of 
design  has  not  been  met  is  lost  if  the  system  has  already 
been  delivered  to  the  warfighter.  The  warfighter  is 
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resolved,  in  fact  trained,  to  make  these  systems  work. 
Testing  early,  and  rigorously  under  precise  and  controlled 
conditions  is  often  a  given.  However  also  factoring  in 
operational  conditions  is  key  towards  ensuring  the  system 
under  development  is  right  for  the  mission. 


Figure  10.  Joint  Vision  2020 

The  Joint  Chiefs  of  Staff  recently  built  off  the  foundation 
established  in  Joint  Vision  2010  and  stated  that  Joint 
Vision  2020  should  consist  of  dedicated  individuals  and 
innovative  organizations  transforming  the  joint  force  for 
the  21st  Century  to  achieve  full  spectrum  dominance  to  be 
persuasive  in  peace,  decisive  in  war,  and  preeminent  in  any 
form  of  conflict.  Several  new  areas  were  highlighted  in  JV 
2020,  including  Joint  Command  and  Control, 
interoperability,  and  Information  Operations. 
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E. 


TEACHING  T&E  &  COMMUNITIES  OF  PRACTICE 
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Figure  11.  Program  Managers  Tool  Kit  -  T&E 

(Version  13) 
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rsity  defines  Test  and 
a  system  or  components 
and  risk  mitigation  and 
and  simulations.  T&E 


permit,  an  assessment  of  the  attainment  of  technical 
performance,  specifications  and  system  maturity  to 
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Table  1 .  T&E  Career  Field  Developmental  Guide 

(1999) 

determine  whether  systems  are  operationally  effective, 
suitable  and  survivable  for  intended  use.  Further,  the 

definition  goes  on  to  describe  two  types  of  T&E  - 

Developmental  (DT&E)  and  Operational  (OT&E) .  The  latest 

release  of  the  Program  Managers  Tool  Kit  has  a  helpful 
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diagram  related  to  Test  and  Evaluation.  This  diagram 
compares  and  contrasts  DT&E  and  OT&E,  and  provides  a 
summary  of  production  qualification  T&E  (sometimes  referred 
to  as  PT&E)  ,  live  fire  T&E  (LFT&E)  and  initial  operational 
T&E  (IOT&E) .  It  also  describes  conditions  when  it  might  be 
prudent  to  combine  DT  and  OT,  and  finally  identifies  the 
T&E  requirements  for  ACAT  I  and  ACAT  II  programs. 

DAU  maintains  a  curriculum  to  ensure  T&E  professionals 
are  given  the  latest  acquisition  information.  Career  field 
developmental  guides  are  available  for  each  acquisition 
field,  and  break  down  paths  for  achieving  certification 
within  each  respective  acquisition  profession,  including 
training,  education,  and  on-the-job  experience.  Similar  to 
other  acquisition  career  fields,  T&E  has  three  levels  of 
proficiency,  with  suggested  competencies  for  each. 

COMOPTEVFOR  OTDG  lists  a  variety  of  helpful  resources, 
including  the  Test  and  Evaluation  Community  Network 
(TECNET,  2003,  n.p.),  which  is  stated  to  include  virtually 
every  testing  resource  the  OTD  will  need,  including 
resources  from  the  other  U.S.  military  services  or  from 
civilian  services,  either  nationally  or  internationally. 

One  of  the  responsibilities  of  the  Deputy  Director, 
Developmental  Test  and  Evaluation  OUSD  (AT&L)  is  to  ensure 
education  and  training  of  the  T&E  workforce,  promote  test  & 
evaluation  best  practices,  and  to  apply  commercial 
practices  to  DOD  programs. 

The  Defense  Test  and  Evaluation  Professional  Institute 
(DTEPI)  serves  as  the  executive  secretary  to  the  T&E 
Functional  Board  for  the  Defense  Acquisition  Workforce 
Improvement  Act  (DAWIA) .  The  Director,  DTEPI  also  chairs 
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both  the  Focus  Group  and  the  Competency  Working  Group.  The 
Focus  Group  is  composed  of  T&E  experts  from  across  the 
career  field  who  develop  competencies,  which  are  the  basis 
of  the  Defense  Acquisition  University  (DAU)  courses  that 
are  required  for  DAWIA  certification  for  the  T&E  functional 
component  of  the  Acquisition  Workforce.  The  Competency 
Working  Group  reviews  the  competencies  developed  by  the 
Focus  Group  and  assigns  a  learning  level  to  each  task. 

DTEPI  is  chartered  by  the  DOD  Office  of  the  Director, 
Operational  Test  and  Evaluation  (DOT&E)  .  The  primary 
purpose  of  the  Institute  is  to  provide  career  and 
professional  development,  education,  training,  and 
recognition  for  the  T&E  professionals  supporting  the  DOD. 
The  Institute  also  is  to  serve  as  a  forum  for  enhancement 
of  the  test  and  evaluation  process  to  meet  current  and 
future  needs  and  challenges. 

As  part  of  the  National  Defense  Authorization  Act  of 
2003  creating  the  Defense  Test  Resource  Management  Center, 
section  234,  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology,  and  Logistics  was  requested  to 
submit  to  Congress  a  report  on  the  capabilities  of  the  test 
and  evaluation  workforce  of  the  Department  of  Defense. 
Working  with  the  Under  Secretary  of  Defense  for  Personnel 
and  Readiness  and  the  Director  of  Operational  Test  and 
Evaluation,  the  following  was  specified  as  requirements  for 
a  comprehensive  plan: 

1)  The  report  shall  contain  a  plan  for 
taking  the  actions  necessary  to  ensure  that  the 
test  and  evaluation  workforce  of  the  Department 
of  Defense  is  of  sufficient  size  and  has  the 
expertise  necessary  to  timely  and  accurately 
identify  issues  of  military  suitability  and 
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effectiveness  of  Department  of  Defense  systems 
through  testing  of  the  systems. 

2)  The  plan  shall  set  forth  objectives  for 
the  size,  composition,  and  qualifications  of  the 
workforce,  and  shall  specify  the  actions 
(including  recruitment,  retention,  and  training) 
and  milestones  for  achieving  the  objectives. 

The  report  needed  to  also  include: 

1)  An  assessment  of  the  changing  size  and 
demographics  of  the  test  and  evaluation 
workforce,  including  the  impact  of  anticipated 
retirements  among  the  most  experienced  personnel 
over  the  period  of  five  fiscal  years  beginning 
with  fiscal  year  2003,  together  with  a  discussion 
of  the  management  actions  necessary  to  address 
the  changes. 

2)  An  assessment  of  the  anticipated 
workloads  and  responsibilities  of  the  test  and 
evaluation  workforce  over  the  period  of  ten 
fiscal  years  beginning  with  fiscal  year  2003, 
together  with  the  number  and  qualifications  of 
military  and  civilian  personnel  necessary  to 
carry  out  such  workloads  and  responsibilities. 

3)  The  Under  Secretary's  specific  plans 
for  using  the  demonstration  authority  provided  in 
section  4308  of  the  National  Defense 
Authorization  Act  for  Fiscal  Year  1996  (Public 
Law  104-106;  10  U.S.C.  1701  note)  and  other 
special  personnel  management  authorities  of  the 
Under  Secretary  to  attract  and  retain  qualified 
personnel  in  the  test  and  evaluation  workforce. 

4)  Any  recommended  legislation  or 
additional  special  authority  that  the  Under 
Secretary  considers  appropriate  for  facilitating 
the  recruitment  and  retention  of  qualified 
personnel  for  the  test  and  evaluation  workforce. 

5)  Any  other  matters  that  are  relevant  to 
the  capabilities  of  the  test  and  evaluation 
workforce . 
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The  OUSD  (AT&L)  response  to  this  request  was  a  report 


to  Congress  entitled,  "Capabilities  of  the  Test  and 
Evaluation  Workforce  of  the  Department  of  Defense." 


Component 

OTAs 

MRTFB 

Other 

Total 

;  Army 

Military 

494 

41 

73 

608 

Civilian 

853 

2,926 

848 

4,627 

Total 
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2,967 

921 

5,235 

Navy/USMC 

Military 

272 

1,310 

30 

1,612 
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mm 
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369 

3,225 

1.831 

Air  Force 

Military 
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3,404 
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4.241 
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3,940 
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739 

7,051 

391 

8,181 

Defense  Agency 

Military 

5 

81 

29 
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16 
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44 

204 

Total 

21 

225 

73 

319 

i  Total,  All  Components 

Military 

BWFETfl 

4,836 

430 

6,576 

Civilian 

1,166 

8,632 

2,786 

12,584 

Total 

2,476 

13,468 

3,216 

19,160 

Table  2.  DOD  T&E  Workforce  by  Component  (2002) 


F.  T&E  BEST  PRACTICES 

In  a  2001  study  sponsored  by  the  Deputy  Director, 
DT&E,  OUSD  (AT&L) ,  conducted  under  contract  by  Science 
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Applications  International  Corporation  (SIAC) ,  a  series  of 
companies  were  visited  to  determine  a  set  of  commercial 
industry  test  and  evaluation  "Best  Practices"  that  may  have 
DOD  test  and  evaluation  organizational  and  process 
applicability.  These  best  practices  were  grouped  under  two 
categories.  Category  I  was  defined  as  best  practices  that 
are  either  easily  implemented  or  have  already  been 
partially  implemented.  Category  II  best  practices  were 
those  less  easy  to  implement  and  requiring  examination  by 
stakeholder  teams  to  determine  feasibility  and  to  develop 
structure  and  schedules.  The  findings  from  this  study 
follow.  Starting  with  Category  I: 

Philosophy,  Policy,  Approach 

1.  Recognize  that  testing  is  a  way  to  identify 
and  solve  problems  early  in  the  process  in 
order  to  control  time,  cost,  and  schedule 
late  in  the  process. 

2.  Recognize  that  best  practices  generate 
success  and  vice  versa. 

Test  Investment 

•  Ensure  early  determination  of  the  investment 
costs  to  acquire  new  capability  for  program 
support . 

•  Require  analytically  sound  ROI  analysis  for 
test  investments. 

•  Ensure  cohesive  (year-to-year)  investment 
plans . 

Test  Execution 

•  Involve  testers  and  evaluators  very  early: 

o  Ensures  testers  know  test  requirements 

o  Ensures  developers  know  requirements 

for  test 

•  Capture  test  costs  at  program  initiation. 

•  Emphasize  concurrent  and  integrated  T&E. 
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•  Institute  formal  quality  check  processes. 

•  Use  System  Integration  Laboratories  and 
embedded  instrumentation. 

•  Give  proper  consideration  to  the  use  of 
external  test  capability  in  test  planning. 

•  Ensure  testers  control  test  planning, 
equipment,  facilities,  instrumentation,  and 
test  resources. 

•  Continue  to  increase  the  use  of  modeling  and 
simulation  to  expand  the  test  process. 

Test  Evaluation 

•  Continue  to  increase  the  use  of  modeling  and 
simulation  to  expand  the  evaluation  context 
based  on  verified  test  data. 

Category  II: 

Philosophy,  Policy,  Approach 


•  Stabilize  corporate  leadership  and  test  staff. 

•  Focus  on  the  quality  of  product  and  process  to  drive 
the  efficiency  and  effectiveness  of  T&E. 

•  Develop  consistent  processes  to  ensure  consistent 
products.  Incorporate  T&E  as  a  process  enabler. 

•  Increase  T&E  to  assure  product  quality  rather  than 
reduce  it  to  save  T&E  cost. 

•  Use  metrics  and  quality  control  processes  to 
understand  how  well  the  test  process  is  operating. 

•  Implement  efficient  and  effective  test  processes  in 
order  to  compete.  Keys: 

o  Ensure  T&E  is  consistently  part  of  the  decision, 
planning,  and  execution  process. 

o  Early  commitment  by  all  stakeholders  on  required 
T&E  resources. 

o  Certification  of  T&E  processes  and  organizations 
(-ISO  9000) 

o  Ensuring  capital  capability. 

Test  Investment 
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•  Charge  cost  of  test  investment  back  to  the  program. 

Test  Execution 

•  Charge  full  cost  of  testing  to  the  program. 

•  Emphasize  multi-use  T&E  platforms. 

•  Do  not  generally  support  the  outsourcing  of  testing 
and  evaluation. 

•  Frequently  use  the  Six  Sigma  or  similar  quality 
processes . 

•  Automate  data  collection  and  archiving. 

•  Benchmark  in-house  and  within  industry. 

•  Use  measurements  and  metrics. 

•  Initiate  programs  to  seek  ten-fold  reductions  in  the 
number  of  software  tests  required. 

•  Integrate  Master  Test  Plans  and  test  execution  with 
program  resources  and  milestones. 

•  Establish  measures  of  effectiveness 

•  Quantify  risk  for  management  decision  when  considering 
reduced  testing. 

•  Train  the  in-house  test  workforce  in  test  engineering 
disciplines . 

Test  Evaluation 


•  Use  Physics  of  Failure  as  a  tool  to  predict  and 

analyze  system  performance  and  shortfalls. 

•  Correlate  faults  and  solutions  in  a  closed  loop 

process  to  ensure  problems  are  resolved. 

Test  Philosophy /Process /Evaluation 

•  Establish  corporate  internal  web  based  sites  for 

exchange  of  ideas,  benchmarks,  data,  applications,  and 
processes.  Address  data  collection  retrieval, 

archiving,  modeling  and  simulation,  and  test  and 

evaluation  methods. 
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The  recommendations  from  this  best  practices  study,  as 
presented  at  the  International  Test  and  Evaluation 
Association  in  November  of  2001  were: 

•  Implement  or  reinforce  the  Category  I  Best  Practices 
in  DOD  as  soon  as  possible. 

•  Develop  implementation  or  reinforcement  strategies  for 
Category  II  Best  Practices  using  DOD  T&E  stakeholder 
teams . 

•  Present  the  results  of  this  study  to  the  DOD 
acquisition  and  T&E  communities. 
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III.  OPEN  SYSTEMS  AND  T&E 


A.  OPEN  SYSTEMS  VERSUS  OPEN  SOURCE 

An  Open  System  (OS)  is  a  system  that  implements 
sufficient  open  specifications  for  interfaces,  services, 
and  supporting  formats  to  enable  properly  engineered 
components  to  be  utilized  across  a  wide  range  of  systems 
with  minimal  changes,  to  interoperate  with  other  components 
on  local  and  remote  systems,  and  to  interact  with  users  in 
a  style  that  facilitates  portability.  An  OS  is 
characterized  by  the  following: 

•  Well-defined,  widely  used,  non-proprietary 
interfaces/protocols . 

•  Use  of  standards  which  are  developed/adopted 
by  industrially  recognized  standards  bodies. 

•  Definition  of  all  aspects  of  system 

interfaces  to  facilitate  new  or  additional 
systems  capabilities  for  a  wide  range  of 
applications . 

•  Explicit  provision  for  expansion  or 

upgrading  through  the  incorporation  of 
additional  or  higher  performance  elements 
with  minimal  impact  on  the  system. 

(IEEE  POSIX  1003.0/D15  as  modified  by  the  Tri-Service  Open 
Systems  Architecture  Working  Group,  2002.) 

The  open  systems  emphasis  in  improved  interfaces  and 
interoperability  provides  opportunity  for  superior 
performance,  accelerated  delivery,  and  more  affordable 
systems.  Open  systems  is  an  "enabler"  for  a  number  of 
acquisition  reform  initiatives  such  as  cost  as  an 
independent  variable,  performance  specifications,  use  of 
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commercial  items,  and  configuration  management  (Open 
Systems  Definition,  2003  11.) 

B.  OPEN  SYSTEMS  AND  STANDARDS 


An  open  system  design  offers  benefits  such  as  life 
cycle  support,  affordability,  and  allowing  timely 
technology  insertion.  However,  there  are  substantial 
differences  in  the  way  open  systems  will  have  to  be  tested 
and  evaluated.  Whereas  open  system  designs  will  rely  on  an 
increased  use  of  commercial  and  non-developmental  items  in 
systems  architectures,  T&E  will  have  to  plan  for 
significant  technical  differences.  These  differences  will 
involve  many  aspects  of  engineering  and  management  such  as 
(DOD  Open  Systems  Joint  Task  Force,  2003,  12)  : 


•  Standards-based  architectures  lessen  the 
degree  of  control  that  DOD  can  expect  to 
exert.  Changes,  fixes,  and  updates  will 
likely  be  under  the  vendor's  control,  but 
adherence  to  the  standard  can  be  expected. 

•  Standards-based  elements  of  the  architecture 
may  be  cheaper  and  faster  to  acquire  but 
will  not  necessarily  be  cheaper  and  faster 
to  integrate,  update,  test,  and  evaluate. 

•  Selection  may  be  risky.  Open  systems 

acquisition  will  demand  that  the  program 
manager  know  substantially  more  about 
technology  and  the  associated  conditions  of 
various  vendors . 


•  Standards 

evolve  with  time.  While 

it 

is  a 

challenge 

to 

visualize 

whether 

a 

given 

standard 

will 

endure. 

it  may 

be 

more 

challenging  to 

determine 

when  to 

swap 

from 

one  standard  to  another. 

•  Integration  becomes  more  important  than 
design.  Performance  requirements  must  be 
realized  without  explicit  control  of  the 
component  design  specification. 


44 


•  Once  integrated,  a  component  may  impact 
global  system  parameters.  Testing  will 
become  an  on-going  and  continuing  activity 
to  verify  that  COTS  and  NDI  items  can  be 
successfully  integrated  into  future  systems. 

The  following  is  defined  in  Volume  1.0  of  the  Navy's 
"DESIGN  GUIDANCE  FOR  THE  NAVY  OPEN  ARCHITECTURE  COMPUTING 
CAPABILITY" (Strei,  2003,  51.3.3) 

An  open  system  approach  has  become  an  important  aspect 
of  system  design  and  development  in  a  wide  variety  of 
enterprises.  This  is  true  primarily  because  open  systems 
convey  certain  benefits  in  terms  of  reduced  life-cycle 
cost,  reduced  time-to-market,  increased  ability  to  inter¬ 
operate  and  cooperate  with  others,  reduced  personnel 
training,  etc.  A  number  of  open  systems  definitions  exist 
within  the  literature.  This  guidance  document  adopts  the 
definition  developed  by  the  DOD  Open  Systems  Joint  Task 
Force  (OSJTF) ,  which  operates  at  the  level  of  the  Office  of 
the  Secretary  of  Defense: 

Open  system:  "A  system  that  implements  sufficient  open 
specifications  for  interfaces,  services,  and  supporting 
formats  to  enable  properly  engineered  components  to  be 
utilized  across  a  wide  range  of  systems  with  minimal 
changes,  to  interoperate  with  other  components  on  local  and 
remote  systems,  and  to  interact  with  users  in  a  style  that 
facilitates  portability."  (DOD  Open  Systems  Joint  Task 
Force,  2003,  52) : 

Open  systems  -  and  architectures  built  to  open  system 
principles  -  possess  a  number  of  common  characteristics. 
While  not  every  open  system  possesses  every  possible 
characteristic,  most  open  systems  tend  to  possess  most  of 
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these  characteristics.  Based  on  examination  of  the  various 
open  system  definitions,  the  attributes  of  an  open  system 
include  the  following: 

•  Use  of  public,  consensus-based  standards 

•  Adoption  of  standard  interfaces 

•  Adoption  of  standard  services  (defined  functions) 

•  Use  of  product  types  supported  by  multiple  vendors 

•  Selection  of  stable  vendor  with  broad  customer  base 
and  large  market  share 

•  Interoperability,  with  minimal  integration 

•  Ease  of  scalability  and  upgradability 

•  Portability  of  application ( s ) 

•  Portability  of  users 

C.  NAVY  OPEN  ARCHITECTURE 

Navy  Open  Architecture  (NOA)  is  an  initiative  to 
design  and  build  a  combat  system  that  meets  changing 
requirements  into  the  21st  century,  while  also  being 
rapidly  and  affordably  upgraded  throughout  its  life  cycle 
(The  Open  Group,  2003) .  NOA  plans  to  adapt  and  exploit  new 
developments  in  open  system  design  principles  and  system 
architectures,  as  well  as  standards-based  computing 
technologies  from  the  Commercial  Off-The-Shelf  (COTS) 
marketplace . 

The  NOA  Working  Group  plans  to  first  develop  a 
coordinated  open  architecture  for  real-time  and  embedded 
system  environments  that  would  be  mutually  beneficial  for 
various  architecture  approaches  to  include  but  not  limited 
to:  DOD  Joint  Integrated  Open  Architectures,  Navy  Open 

Architecture,  Air  Force  Viable  Combat  Aircraft  Joint 
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Council  on  Aging  Avionics,  Modular  Open  Systems  Approach 
Interoperability  Initiative,  Army  Weapon  Systems  Common 
Operating  Environment  and  various  open  architectures  from 
corporations  and  system  integrators. 


PROCESSING  SYSTEM  ARCHITECTURE  EVOLUTION 

COMPUTER  ARCHITECTURE  EVOLUTION 
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Figure  12.  Computer  Processing  Architecture 

Evolution 

This  approach  would  leverage  the  information 
technology  industry  investment  in  the  development  of  COTS 
components.  The  use  of  COTS  should  allow  for  easy 
transition  to  commercially  available  advanced  hardware  and 
software  technology.  The  key  to  this  approach  however,  is 
the  use  of  COTS  products  that  already  use  open  standards. 
Open  standards  promote  conformance  at  the  interface  level 
ensuring  compatibility  and  interoperability.  Development 
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of  a  fully  open  architecture  would  allow  for  the  use  of 
future  technology  and  transition  the  insertion  of 
components  from  one  generation  to  the  next  based  on 
hardware  and  software  products  that  are  conformant  to  open 
standards  (Chief  Information  Office,  The  Open  Group,  2003.) 

The  Navy  gradually  wants  to  do  away  with  decades-old 
proprietary  combat-system  software  and  replace  it  with  a 
modern  open  architecture  (Erwin,  2003,  11.)  The  cost  to 

move  MILSPEC,  Navy-specific  systems  into  commercial 
computing  environments  is  difficult  to  calculate,  but  could 
save  billions  of  dollars,  over  time.  And  an  open 
architecture  could  help  the  Navy  improve  the  capabilities 
of  the  AEGIS  combat  system  for  future  missile-defense 
missions . 


An  open  architecture  is  what  technologists 
call  a  "plug  and  play"  computing  environment,  one 
that  allows  for  easy  upgrades  of  software 
applications,  without  having  to  reengineer  a 
warship's  entire  combat  system.  "The  analogy  is 
that  when  you  get  a  new  refrigerator,  you  don't 
need  to  worry  about  testing  the  sink  and 
everything  else,"  said  one  industry  expert 
(Erwin,  2003,  12 . ) 

Whereas  the  Navy  already  spends  billions  of  dollars 
annually  upgrading  proprietary  software,  an  open 
architecture  is  seen  as  the  path  to  significant  savings. 

There  is  undoubtedly  a  technological 
incentive  surrounding  open  architecture.  While 
pushing  forward  in  such  pursuits,  the  Navy  must 
simultaneously  provide  new  combat-system 
computers,  while  keeping  the  AEGIS  fleet  ready 
for  war  and  meeting  its  missile-defense 
commitments.  By  2005,  the  Navy  is  expected  to 
deploy  18  anti-ballistic  missile  AEGIS  warships— 
three  cruisers  and  15  destroyers  (Erwin,  2003, 

13  .  ) 
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In  charge  of  developing  a  plan  to  introduce  open- 
architecture  computers  in  the  Navy  by  2010  is  a  new 
organization  created  last  year,  the  Program  Executive 
Office  for  Integrated  Warfare  Systems  (PEO  IWS) .  PEO  IWS  is 
working  with  large  and  small  companies  to  develop  standards 
and  protocols  that  will  eventually  influence  every  computer 
system  in  the  Navy. 

Open  architecture  is  "the  right  way  to  go" 
for  the  Navy,  said  Rear  Adm.  C.  Tom  Bush,  the 
head  of  PEO  IWS.  "We  need  to  stop  building 
proprietary  architectures."  Bush  believes  that  an 
open  architecture  can  help  make  those  upgrades 
easier  and  less  costly,  saving  the  Navy  at  least 
$1  billion  a  year  (about  50  percent  of  the 

service's  annual  expenditures  on  software 
upgrades)  .  "On  a  good  day,  when  something  needs 
an  upgrade,  it  requires  seven  to  28  changes," 

Bush  said.  "We  can't  build  a  combat  system  for 
every  ship.  But  we  can  build  a  single 
architecture."  (Erwin,  2003,  55-7.) 

Another  benefit  of  open  architecture  is 

"interoperability,"  said  Rear  Adm.  Henry  G. 
Ulrich  III,  director  of  Naval  Surface  Warfare. 

"The  business  model  and  the  architecture  we  have 
now  are  driving  us  away  from  interoperability, 
not  only  with  our  allies  but  amongst  ourselves. 

. . .  The  current  technology,  which  is  not 

compatible  with  anything  else,  drives  up  the  cost 
of  producing  and  upgrading  software." 

The  Navy' s  director  of  open  architecture  is 
Tom  Pendergraft,  a  career  combat-system  engineer. 

Upon  taking  the  job  only  a  few  months  ago,  he 
quickly  learned  that  bringing  open  architecture 
to  the  Navy  is  less  about  technology  and 

engineering  than  about  "culture  and  business 
models  changing,"  he  said  in  an  interview.  The 
enormous  expense  associated  with  software 

upgrades  and  a  desire  to  improve  the  current 
technology  make  it  imperative  for  the  Navy  to 
begin  migrating  to  open  architecture,  he  said. 
Upgrading  the  AEGIS  combat  system  on  average  can 
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cost  hundreds  of  millions  of  dollars.  Not  only 
can  the  service  not  afford  these  prices,  but  in 
many  cases,  the  computers  have  been  upgraded  so 
much  already  that  their  capacity  to  grow  has  been 
exhausted.  "Necessity  is  the  mother  of 
invention,"  Pendergraft  said.  "That  is  what  we 
are  talking  about  here." 

At  the  center  of  the  Navy' s  theater  defense  capability 
is  the  AEGIS  Weapon  System.  With  a  phased-array  radar  that 
can  track  hundreds  of  targets  simultaneously,  and  a  command 
and  control  computer  system  allowing  simultaneous  tracking 
operations  in  air,  surface  and  undersea  warfare,  keeping 
pace  with  the  emerging  threat  is  critical.  Detect,  track 
and  engage  functions  require  enormous  computing  capability, 
with  millions  of  actions  being  performed  by  the  host  weapon 
system  every  second. 

In  the  early  days  of  AEGIS,  said 
Pendergraft,  "we  had  a  single  computer  that  did 
all  the  computation  for  warfare  systems."  As  the 
operations  became  more  complex,  when  the  Navy 
started  using  more  advanced  weapons,  the 
computing  power  needs  grew  exponentially. 
Another  drawback  to  the  current  technology  is 
that  it  is  "serial,"  meaning  it  can  do  one  thing 
at  a  time  -  detection,  tracking,  identification, 
decision,  engagement.  "You  only  had  one  computer 
to  do  all  that.  ...  Our  architecture  is  serially 
based,  with  point-to-point  connectivity, "  said 
Pendergraft.  "Pretty  soon,  you  have  what  we  call 
spaghetti  code."  (Erwin,  2003,  59.) 

This  "spaghetti  code"  is  commonly  the  reason  why 
upgrades  are  so  expensive.  Besides  the  fact  that  this 
legacy  code  can  only  be  maintained  by  a  shrinking  resource 
pool,  when  a  single  applications  is  upgraded,  whether  to 
fix  a  software  bug,  or  to  build  in  more  capability,  the 
entire  weapon  system  has  to  be  tested  to  make  sure  no 
changes  were  made  inadvertently  to  other  functions. 
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"Today,  when  we  make  a  change,  by  the  nature 
of  the  shared-memory  architecture,  you  end  up 
having  to  retouch  the  entire  system, "  said 
Orlando  Carvalho,  vice  president  of  AEGIS 
programs  at  Lockheed  Martin  Corp.  "In  some  cases, 
you  have  to  make  many  changes  for  a  fairly  small 
upgrade."  Norm  Malnak,  Lockheed  Martin  technical 
director,  said  the  problem  is  exacerbated  by  the 
presence  of  multiple  AEGIS  baselines  (software 
releases)  throughout  the  fleet.  The  oldest  ships, 
for  example,  use  baseline  1.4.  Others  have 

baselines  2.1,  5.3  or  6.3,  for  the  newest  ships. 

Lockheed  Martin  is  developing  baseline  7.1,  with 
more  advanced  features.  An  open  architecture  will 
help  "get  commonality  across  the  fleet, "  said 

Malnak.  "That  saves  a  lot  of  money."  (Erwin, 

2003,  111.) 

The  Navy  has  spent  hundreds  of  millions  of 
dollars  on  modern  computers  to  expand  the  memory 
and  computer  processing  speed  of  AEGIS  combat 

applications,  but  fast  PCs  is  not  what  open 
architecture  is  about,  explained  Pendergraft.  "We 
went  to  COTS  computers,  but  we  haven't  done 

anything  with  all  the  point-to-point 

connectivity.  .  .  .  With  open  architecture,  we  are 
changing  the  fundamental  structure." 

Navy  Standard  Computers  are  used  to  process  the  input 
and  output  data.  AEGIS  uses  several  variants  of  the 
AN/UYQ-43  computer,  but  each  lacks  the  speed  and  memory  of 
personal  computers  found  commonly  in  the  office  or  in  the 
home.  The  Navy  has  increased  computing  power  through  the 
use  of  adjunct  processors  and  additional  memory,  but  the 
UYQ-43  handles  the  critical  functionality  that  eventually 
builds  fire  control  solutions,  leading  to  ordnance  on 
target . 


"Our  current  Navy  Standard  Computers  are  at 
about  99  percent  capacity,"  said  Pendergraft. 
"Every  time  we  want  to  add  a  new  function,  we 
can't  do  it  on  NSC,  so  we  add  adjunct  computers." 
This  setup  still  maintains  the  "spaghetti  code" 
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structure.  By  adding  more  processors  and 
functions,  "all  the  stuff  starts  crisscrossing. 

We  have  point-to-point  spaghetti  code  all  over 
the  place.  It  makes  it  very  complicated  and 
expensive  to  maintain."  Further,  "we  are 
prohibited  right  now  from  adding  a  lot  of 
significant  war-fighting  capability,  because  we 
don't  have  the  computing  capacity,"  he  said. 

To  make  the  open  architecture  plan  work,  "we 
have  to  stop  people  from  putting  proprietary 
computers  on  ships.  That  is  what  kills  us.  Every 
time  there  is  an  upgrade  or  the  manufacturer  goes 
out  of  business,  we  are  toast.  We  have  to  hire 
someone  to  rebuild  that  system,  or  we  have  to 
keep  someone  in  business  for  a  lot  longer  than  we 
want  to."  Unfortunately,  he  said,  "There  is  no 
police  force  for  specs  and  standards."  (Erwin, 

2003,  113.) 

Open  architecture  requires  a  significant  up-front 
investment  and  questions  about  its  claimed  merits.  PEO  has 
provided  various  estimates  of  what  it  would  cost  to  convert 
the  entire  fleet,  but  the  debate  over  legacy  development 
versus  open  systems  development  seems  to  be  winding  down. 

D.  OPEN  SYSTEMS  AND  MDA 

As  the  Navy  moves  closer  to  fielding  a  ballistic- 
missile  defense  for  the  United  States,  the  ability  of 
computers  to  do  the  job  comes  into  question.  Even  in  the 
most  modern  of  deployed  weapon  systems,  the  computing 
environment  is  taxed  to  the  point  that  further  enhancement 
or  upgrades  may  not  be  possible.  According  to  PEO  IWS's  OA 
lead,  Tom  Pendergraft,  "current  computing  plants  are  pretty 
full.  If  you  want  to  add  BMD  on  top  of  that,  that  is  going 
to  be  pretty  tough.  ...  If  we  go  to  open  architecture,  with 
distributed  computing,  we  would  have  virtually  unlimited 
[computational  power]  resources."  (Erwin,  2003,  517.) 
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It  is  possible  to  accomplish  missile-defense 
missions  in  legacy  AEGIS  ships,  "assuming  some 
modifications  to  open  up  the  architecture,"  he 
said.  "You  are  not  going  to  get  there  with  one 
computer . " 

Future  missile-defense  capabilities  the 
Pentagon  envisions,  such  as  new  solid-state  radar 
and  extended  range  weapons,  will  require  more 
computing  power  than  currently  exists.  "As  you 
move  forward  with  missile  defense,  you  want 
additional  signal  processing  capability, "  said 
Chris  Myers,  director  of  missile  defense  and 
radar  programs  at  Lockheed  Martin.  The  company  is 
responsible  for  various  pieces  of  the  naval 
missile-defense  program,  including  AEGIS,  cruiser 
upgrades,  and  the  development  of  an  active  solid- 
state  radar. 

We  want  to  upgrade  those  computing  plants  so 
it  makes  it  a  lot  easier  to  upgrade  AEGIS,"  said 
Myers.  "In  the  future,  you  want  to  see  targets 
further  away,  smaller  things,  you  need  additional 
radar  power  and  sensitivity  to  see  that.  ... 

There  is  additional  computing  power  required  as 
you  move  to  a  solid-state  radar.  (Erwin,  2003, 

519.  ) 

Lockheed  is  one  of  4  9  companies  that 
received  contracts  to  help  the  Navy  come  up  with 
commercial  standards  and  protocols  for  the  open 
architecture.  The  plan  is  to  begin  installing  the 
new  technology  on  surface  ships  and  then  expand 
to  submarines.  The  DDX  land-attack  destroyer,  to 
be  deployed  by  2012,  is  expected  to  be  the  Navy's 
first  truly  "open-systems"  ship. 

In  March,  PEO  IWS  released  an  interim  set  of 
specifications  and  standards  that  new  programs  will  have  to 
follow,  in  order  to  be  open-architecture  compliant. 
Existing  legacy  systems  however,  will  have  to  be  addressed 
as  well,  but  there  are  challenges.  Existing  weapons  systems 
using  dated  technology  require  very  specific  techniques  and 
talents  to  upgrade,  which  will  likely  be  very  expensive.  In 
addition,  DON  continues  to  train  and  fight  wars  with  ships 
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that  cannot  afford  to  be  taken  out  of  service  for  the  time 
it  would  take  to  complete  a  comprehensive  upgrade,  let 
alone  test  and  evaluate.  Moving  into  OA  standards  allows 
future  systems  to  evolve,  but  will  also  allow  existing 
systems  to  eventually  become  more  open. 

A  transition  to  open  architecture,  for 
example,  would  involve  "taking  pieces  of  our 
combat  systems,  throwing  away  the  old  code,  rack 
and  stack  the  algorithms,  write  them  in  modern 
computing  program  so  they  can  run  on  a  modern 
computing  environment,"  said  Pendergraft.  "In  the 
AEGIS  program,  we  are  starting  now  to  open  up  the 
system,"  he  said. 

Rick  Scharadin,  program  manager  at  Lockheed 
Martin,  said  the  company  will  demonstrate  how 
segments  of  AEGIS  can  be  converted  to  open 
architecture  in  a  piecemeal  fashion.  The  first 
step  is  to  upgrade  the  computing  environment  for 
the  radar,  by  2006.  The  second  piece  is  to 

convert  the  displays,  by  2007.  In  2008,  the  plan 
is  to  demonstrate  open-architecture  radar  and 
displays,  weapons  control  and  fire  control.  "The 
key  is  to  find  those  parts  that  you  could  easily 
remove,"  said  Pendergraft.  "The  only  way  we' 11  be 
able  to  do  this  is  one  part  at  a  time.  Can't  do 
it  all  at  once."  (Erwin,  2003,  513-21.) 

PEO  IWS  plans  to  spend  about  $50  million  a 
year  on  research  related  to  open  architecture.  A 
lot  more  money,  however,  will  be  needed  to 

upgrade  ships.  Those  funds  may  have  to  come  from 
ongoing  acquisition  programs,  a  prospect  that 
Pendergraft  acknowledged  will  stir  the  proverbial 
"rice  bowls"  associated  with  military  projects. 

"Some  of  the  programs  of  record  are  going  to  have 
to  change  direction  in  order  to  pay  for  this," 
said  Pendergraft. 

"In  a  front-line  combat  system  like  AEGIS, 
you  cannot  do  plug  and  play  without  doing 

specific  reengineering  to  make  sure  you  haven't 
contaminated  the  system  integrity, "  said  retired 
Rear  Adm.  George  Meinig,  who  was  the  AEGIS 

technical  director  in  the  mid-1980s.  "The 
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are 


benefits  of  open  architecture  are  still 

desirable,"  he  said.  "But  there  is  no  assurance 

that  unaltered  commercial  products  can  meet  the 
performance  requirements  of  the  combat  system. 

.  .  .  You  have  to  do  careful  testing  to  make  sure 
the  design  is  compliant  with  the  requirement  and 
that  you  haven't  messed  up  the  whole  system." 
(Erwin,  2003,  525.) 

As  the  benefits  from  employing  open  architectures  are 
becoming  more  well  understood,  the  return  on  investment 

might  not  be  seen  for  many  years.  And  the  initial 

conversion  to  an  OA  would  not  necessarily  increase  the 
capability  right  away,  but  instead  allow  for  the  potential 
growth  in  the  future.  So  as  current,  in-service  weapon 


systems  are  on  the  verge  of  obsolescence,  open  systems 
architecture  could,  in  concept,  give  them  the  ability  to 
serve  the  warfighter  into  the  future. 


According  to  some  very  recent  educational  forums 
sponsored  by  DAU  and  focused  on  Program  Management  looking 
towards  the  future,  open  architecture  can  be  summarized 
with  the  following  aspects: 


•  Today' s  Fleet  computing  architectures  are  performance 
limited  and  expensive  to  upgrade. 

•  Implementation  of  warfighting  functions  using  standards- 
based  solutions  will  enable  common,  interoperable 
capabilities  to  be  fielded  faster  at  reduced  cost. 

•  Rapid  Technology  Insertion  Program  (RTIP)  will  provide  a 
structured  approach  for  introduction  of  OA  components 
into  the  Fleet  (Program  Managers  Workshop,  2003.) 


E.  NOA  -  EMERGING  GUIDANCE 


From  Version  1.0  of  DESIGN  GUIDANCE  FOR  THE  NAVY  OPEN 
ARCHITECTURE  COMPUTING  CAPABILITY,  Navy  open  architecture 
is  the  high-level  technical  structure  of  the  weapon  system 
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as  designed  in  accordance  with  the  principles  of  open 
systems  to  achieve  both  real-time  mission  requirements  and 
life-cycle  supportability  goals.  Technical  characteristics 
of  NOA  include: 

•  Distribution  of  processing 

•  Widespread  use  of  standards-based  COTS  computing 
technologies 

•  Functional  capabilities  implemented  as  medium- 
grain  components 

•  Use  of  object  oriented  (00)  programming  within 
components  and  middleware  technologies  for 
interconnection  of  and  interoperation  among 
components 

•  Use  of  design  mechanisms  such  as  client-server  to 
maximize  isolation  of  implementation  details  from 
publicly  visible  services  and  APIs 

•  Portability  and  transparency  of  application 
components  with  respect  to  physical  location  and 
network,  processor  and  operating  system  types, 
etc . 

The  corresponding  goals  of  the  NOA  are  to  provide  to 
the  weapon  system  not  only  the  benefits  of  assured 
technical  performance,  but  also  of  reduced  life-cycle  cost, 
affordable  technology  refresh,  and  reduced  upgrade  cycle 
time.  Expected  benefits  include: 

•  Scalable,  load  invariant  performance 

•  Enhanced  information  access  and  interoperability 

•  Enhanced  system  flexibility  for  accomplishment  of 
mission  and  operational  objectives 

•  Enhanced  survivability  and  availability 
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•  Reduced  life-cycle  cost  and  affordable  COTS 
technology  refresh 

•  Reduced  cycle  time  for  changes  and  upgrades. 


Navy  OA  Functional  Architecture.. .Proposed  End  State  View 
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Figure  13.  Notional  NOA  Functional 
Architecture 


F.  NOA  GOALS  AND  FUNCTIONAL  ARCHITECTURE 

In  broad  and  general  terms  (Strei,  2002),  architecture 
is  defined  as  "the  structural  design  of  an  entity."  Adding 
"openness"  to  the  list  of  architectural  characteristics 
implies  that  the  "structure"  of  the  architecture  explicitly 
promotes  interoperability,  both  internally  and  externally, 
as  well  as  ease  of  modification  and  extension. 


57 


It  is  an  engineering  truism  that  what  is 
achievable  in  system  design  (architecture)  is  a 
function  of  not  only  the  task  to  be  accomplished 
but  also  the  technologies  that  are  available. 
However,  the  evolution  of  high  performance  COTS, 
combined  with  continued  growth  of  weapon  system 
and  combat  system  requirements,  provides  an 
opportunity  to  design  an  architecture  more 
capable  of  exploiting  new  technologies  than  the 
federated  legacy  architecture  that  has  served  the 
Navy  for  well  over  two  decades.  The  need  for 
evolution  toward  an  open  architecture  is 
motivated  by  both  performance  and  supportability 
considerations.  Commensurate  with  this  dual  set 
of  motivating  factors,  the  goals  of  the  NOA  are 
as  follows: 

•  Combat  system,  weapon  system,  command  support 
system  and  HM&E  capabilities  that  continue  to 
pace  the  threat . 

•  System  design  that  fosters  affordable 
development  and  life-cycle  maintenance. 

•  System  design  that  reduces  upgrade  cycle  time 
and  time-to-deployment  for  new  features. 

•  Architecture  that  is  technology  refreshable 
despite  rapid  COTS  obsolescence. 

•  Improvements  in  NWS  Human  Systems  Integration. 

Finally,  system  requirements  may  include  not 
only  capability  and  performance  goals  but  also 

non-functional  engineering  goals  as  well  ("- 
ilities") .  In  addition  to  traditional  metrics 
such  as  reliability  and  survivability,  NOA 

metrics  include  qualitative  goals  such  as 
portability,  scalability,  extensibility,  and 

flexibility  of  use.  These  goals  will  be  met,  in 
part,  by  careful  design  and  in  part  through  use 
of  open  systems  principles  and  standards.  This 
document  focuses  primarily  on  the  technical 
aspects  of  designing  NOA.  However,  in  many 

cases,  the  recommended  design  choices  and 

technologies  are  chosen  with  the  goal  of 
supporting  this  qualitative  metrics  as  well. 
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G.  WHY  OPEN  ARCHITECTURE? 

In  a  recent  Defense  News  article  discussing  the  U.S. 
Navy' s  decision  to  change  its  acquisition  strategy  for  CEC, 
replacing  a  winner-take-all  approach  with  a  series  of 
smaller  competitions,  the  rationale  was  OA. 

"We're  totally  changing  the  plans  for  CEC  Block 
2,"  said  a  Navy  official.  Instead  of  a  closed 
system.  Block  2  will  incorporate  open- 
architecture  standards  to  hold  costs  down,  allow 
more  joint  interactions,  and  help  the  fleet  to 
adapt  more  quickly  to  new  threats.  Service 
officials  hope  this  new  approach  will  encourage 
innovation,  entice  more  firms  to  compete  for  the 
work,  and  ultimately  push  down  purchase  costs 
(Sherman,  p.l,  2003.) 

Open  architecture  is  the  key  to  affordable  21st 
Century  joint  combat  capability  (Program  Managers  Workshop, 
2003.)  OA  enables  current  weapon  systems,  and 
corresponding  computing  systems,  which  today  cannot  support 
emerging  Sea  Power  21  warfighting  capability  requirements, 
to  be  upgradeable  to  meet  these  future  needs.  In  addition, 
OA  is  claimed  to  be  affordable,  although  this  is  a  subject 
of  great  debate  at  present.  What  is  not  debatable  is  that 
current,  in-service  computing  architectures  are 
unaffordable.  Presently,  each  ship  class  addresses  common 
problems  uniquely,  while  software  and  hardware  changes  are 
interdependent.  Finally,  OA  must  support  Joint 
Interoperability.  Today's  existing  in-service 
architectures  cannot  support  Forcenet  implementation. 

The  OA  implementation  strategy  must  include  several 
concepts.  To  begin  with,  computer  program  upgrades  that 
provide  only  marginal  warfighting  capability  enhancement 
must  be  frozen  and  future  upgrade  plans  terminated.  OA 
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technical  and  functional  architectures  must  be  completed 
and  consensus  gained  for  scaleable.  Navy-wide  applications. 
A  Rapid  Technology  Insertion  Program  process  has  been 
proposed  to  transition  promising  technologies  to  certified 
warfighting  products.  All  new  systems  must  comply  with 
agreed-upon  and  documented  OA  standards  specifications  and 
guidance.  Finally,  proponents  of  open  architecture  should 
pursue  coordination  and  agreements  with  all  other 
initiatives  and  programs. 
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IV.  T&E  IN  THE  FUTURE 


A.  INDICATORS 

In  an  Inside  the  Navy  article  dated  September  8  2003, 
entitled,  "Study  prepared  for  Young:  Cohen  predicts  Navy 
will  put  EM  gun  on  DD  (X)  within  a  decade,"  The  navy  plans 
to  have  an  electromagnetic  rail  gun  aboard  a  DD  (X) 
destroyer  in  eight  to  10  years,  according  to  Chief  of  Naval 
Research  Rear  Admiral  Jay  Cohen,  who  expects  the  total  cost 
of  developing  one  or  two  gun  prototypes  would  be  $500 
million  to  $1  billion.  Though  not  explosive,  EM  gun 
projectiles  would  hit  targets  with  uncanny  speed  and 
devastating  force,  setting  a  new  standard  for  deadly,  long- 
range  shipboard  guns.  "I  will  tell  you,  we  think  in  eight 
to  10  years,  you're  going  to  see  a  250-to-300  mile 
electromagnetic  rail  gun  on  DD (X) , "  Cohen  predicted  in  a 
presentation  at  "COMDEF  2003"  in  Washington,  DC. 

"We  must  acknowledge  that  our  way  of  war  requires  superiority  in  all 
mediums  of  conflict,  including  space.  Thus,  we  must  plan  for  and 
execute  to  win  space  superiority." 

Gen.  Richard  B.  Myer,  Vice-Chairman,  U.S.  Joint  Chiefs  of  Staff 

"The  idea  of  putting  weapons  in  space  to  dominate  the  globe  is 
simply  not  compatable  with  who  we  are  and  what  we  represent  as 
Americans.” 


Figure  14.  From  NMD:  The  Arctic  Dimension 

(2003) 

While  there  is  plenty  of  speculation  and  high  hopes 
for  future  systems  to  be  developed  and  acquired  in  the 
future,  but  there  is  very  little  evidence  of  how  we  might 
prepare  to  test  and  evaluate  these  emerging  systems.  In 
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addition,  there  appears  to  be  no  consensus  about  what 
testing  in  the  future  will  look  like. 

B.  HOW  DOES  OPEN  SYSTEMS  ARCHITECTURE  CHANGE  THINGS? 

In  a  speech  by  RADM  Phillip  M.  Balisle,  Director, 
Surface  Warfare  Division  (N76)  to  the  Surface  Navy 

Association  (March  2002)  he  stated,  "As  we  explore  the 
transformation  of  the  existing  AEGIS  baselines  into  an  open 
architecture,  distributed  processing  combat  system,  we 
intend  to  build  these  interoperability  enhancements  into 
our  new  systems  from  the  ground  up."  He  continued  by 

saying,  "Following  the  successful  transition  to  a  complete 
COTS  computing  environment  on  our  new  construction  AEGIS 

DDGs,  AEGIS  baseline  development  will  introduce  an  open 
architecture,  high  performance,  interoperable  and  network 
ready  software  architecture,  which  will  eliminate  many  of 
the  interoperability  limitations  of  today' s  combat 
systems."  He  concluded  his  speech  by  stating,  "When  we 
align  our  systems  and  integrate  them  using  a  systems 
engineering  approach  into  a  new  architecture  which  allows 
for  the  efficient  exchange  of  required  data  across  the 
network,  we  will  realize  another  dramatic  increase  in 

situational  awareness,  speed  of  command  and  synchronization 
that  will  buy  back  even  more  critical  battle  space  for  our 
Warfighters  .  " 

C.  T&E  IN  THE  MISSILE  DEFENSE  AGENCY 

The  Missile  Defense  Agency,  in  July  2003,  released  the 
following  information  regarding  test  and  evaluation  (BMD 
Test  &  Evaluation,  2003,  pp .  1-2.): 

The  Ballistic  Missile  Defense  (BMD)  test 

philosophy  recognizes  the  need  for  an  integrated. 
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phased  test  program  that  comprehensively  covers 
all  facets  of  testing.  Testing  components, 
subsystems  and  systems,  especially  early  in  the 
developmental  cycle,  can  determine  current 
performance  capabilities  and  identify  potential 
design  areas  where  technology  can  increase 
overall  system  capability.  Later  testing 
demonstrates  and  measures  the  effectiveness  and 
suitability  of  missile  defense  systems  in  their 
intended  operational  environments. 

The  BMD  System  (BMDS)  test  methodology  adds 
system  complexities  over  time.  For  example, 
system  performance  in  the  presence  of 
countermeasures  and  operations  in  increasingly 
stressful  combat  scenarios  would  be  addressed  in 
segmental  tests.  This  step-by-step  approach 
facilitates  timely  assessments  of  the  most 
critical  design  risk  areas. 

The  MDA  test  and  assessment  program  supports 
credible  decisions  with  respect  to  the  BMDS  and 
its  elements.  Specific  program  objectives  focus 
on:  characterizing,  demonstrating,  measuring  and 
verifying  achievement  of  BMDS  capabilities; 
executing  BMDS  test  events;  facilitating  credible 
testing  of  BMDS  Elements,  technology  experiments 
and  international  collaborative  programs;  and 
anchoring  Modeling  and  Simulation  with  test  data 
for  use  in  measuring  performance  throughout  the 
test  envelope. 

Meeting  the  challenges  of  BMD  testing 
requires  an  extensive  test  infrastructure. 
Collectively,  groundtest  facilities,  ranges, 
sensors  and  instrumentation  assets  provide 
valuable  BMD  program-wide  risk  reduction  and  test 
capability  to  assess  BMD  system  and  element 
performance.  Ground  tests  are  conducted  at  high¬ 
speed  sled  tracks,  hardware-in-the-loop 
facilities,  aero-ballistic  ranges,  aero-optic  and 
aero-thermal  shock  tunnels  and  space  chambers. 

MDA  deploys  mobile  airborne  sensors  to 
ranges  during  flight  tests,  which  have  onboard 
signal  and  data  processing  and  collection 
capabilities.  More  recently  this  includes  the 
development  of  transportable  instrumentation  and 
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common  standards  to  support  MDA  testing  with 
flexible  scenarios  at  a  variety  of  locations. 

MDA  conducts  BMDS  Integrated  Tests  using 
selected  hardware  and  software  from  the 
individual  elements  to  investigate  performance, 
joint  operations  and  interoperability.  These 
tests  include  the  Critical  Measurements  Programs 
and  the  System  Integration  Tests.  The  former  are 
live  test  flights  that  provide  common  data 
collection  opportunities  and  the  latter  are  live 
intercept  tests  involving  representations  of 
potential  future  threats.  Results  of  all  tests 
are  used  to  conduct  annual  system-  wide 
capability  assessments. 

D.  THE  FATE  OF  OT  -  ONE  EXAMPLE:  MDA 

The  Bush  administration  is  proposing  to  exempt  the 
Pentagon's  controversial  missile  defense  system  from 
operational  testing  legally  required  of  every  new  weapons 
system  in  order  to  deploy  it  by  2004  (Schrader,  2003,  p.l.) 

In  the  FY  2004  budget,  is  a  request  to 
rewrite  a  law  designed  to  prevent  the  production 
and  fielding  of  weapons  systems  that  don't  work. 

If  the  provision  is  enacted,  it  would  be  the 
first  time  a  major  weapons  system  was  formally 
exempted  from  the  testing  requirement.  The 
proposal  follows  administration  moves  to  bypass 
congressional  reporting  and  oversight 
requirements  in  order  to  accelerate  development 
of  a  national  missile  defense  system.  One  of 
Bush's  goals  when  he  took  office  was  to  carry  out 
a  missile  defense  system  —  an  idea  first  proposed 
by  President  Reagan  —  and  he  almost  immediately 
expanded  the  scope  and  the  funding  of  the 
controversial  program,  which  had  encountered 
scientific  and  budgetary  difficulties  in  recent 
years . 

Last  year,  to  help  achieve  that  goal. 
Defense  Secretary  Donald  H.  Rumsfeld  gave  the 
Missile  Defense  Agency  unprecedented  managerial 
autonomy  and  removed  procurement  procedures  that 
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were  intended  to  ensure  new  weapons  programs 
remain  on  track  and  within  budget  (Schrader, 
2003,  p . 2 . ) 

While  the  exemptions  granted  previously  gave 
the  missile  defense  program  an  unprecedented 
degree  of  autonomy  from  congressional  oversight, 
they  did  not  exclude  it  from  testing. 
Highlighting  its  technical  weaknesses  has  been 
opponents'  best  hope  for  slowing  the  long-debated 
program.  In  recent  years,  critics  repeatedly 
have  used  Pentagon  data  from  missile  defense 
flight  tests  to  challenge  whether  the  experiments 
were  as  successful  as  claimed. 

The  latest  proposal  from  the  Pentagon  would 
exempt  the  missile  defense  deployment  from  a  law 
that  requires  the  Defense  Department  to  certify 
that  appropriate  operational  testing  has  been 
completed  before  putting  systems  into  production. 

The  Bush  administration  announced  in 
December  2002  a  goal  of  having  a  limited  ground- 
based  system  operational  in  Alaska  and  at 
Vandenberg  Air  Force  Base  in  California  by  Oct. 
1,  2004  (Schrader,  2003,  p.3.)  "The  moves  last 

year  were  just  about  reporting  requirements. 
This  is  different, "  said  Philip  Coyle,  director 
of  operational  testing  and  evaluation  for  the 
Pentagon  from  1994  to  2001.  "This  is  about 
obeying  the  law.  Without  these  tests,  we  may 
never  know  whether  this  system  works  or  not,  and 
if  they  are  done  after  this  system  is  deployed, 
we  won't  know  until  we've  spent  $70  billion  on  a 
ground-based  missile  defense  system." 

In  a  letter  to  Rumsfeld,  Feinstein  wrote:  "I 
believe  that  any  deployed  missile  defense  system 
must  meet  the  same  requirements  and  standards 
that  we  set  for  all  other  fully  operational 
weapons  systems.  Indeed,  given  the  potential 
cost  of  a  failure  of  missile  defense,  I  believe 
that,  if  anything,  it  should  be  required  to  meet 
more  stringent  test  standards  than  normally 
required . " 

Rumsfeld  replied  that  an  exemption  made 
sense  in  the  case  of  missile  defense  (Schrader, 
2003,  p.4.)  "I  happen  to  think  that  thinking  we 
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cannot  deploy  something  .  .  .  until  you  have 
everything  perfect,  every  '  i '  dotted  and  every 
'  t '  crossed,  it's  probably  not  a  good  idea,"  he 
said.  "In  the  case  of  missile  defense,  I  think 
we  need  to  get  something  out  there,  in  the 
ground,  at  sea,  and  in  a  way  that  we  can  test  it, 
we  can  look  at  it,  we  can  develop  it,  we  can 
evolve  it,  and  . . .  learn  from  the  experimentation 
with  it . " 

Rumsfeld  pointed  out  that  two  other  weapon 
systems  in  recent  years  —  the  Predator  unmanned 
aerial  vehicle  and  the  Joint-STAR  aircraft  radar 
systems  —  were  deployed  before  they  were  tested 
operationally.  But  those  systems  did  eventually 
go  through  operational  testing,  and  neither  went 
into  full  production  until  the  testing  was 
completed.  There  is  no  guarantee  the  operational 
testing  will  ever  take  place  if  the  law  is 
changed  to  allow  the  system  to  be  deployed 
(Schrader,  2003,  p.5.) 

E.  WHERE  MODERN  T&E  MUST  EVOLVE 

To  meet  the  challenges  presented  by  an  evolutionary 
acquisition,  and  a  US  Navy  deep  into  transformation, 
requires  a  T&E  methodology  that  is  equally  as 
transformational . 

The  Navy  is  actively  engaged  in  the  acquisition  of 
future,  technology-exploiting  weapon  systems.  In  a  recent 
article  (Schweizer,  2003,  p.5)  discussing  the  merits  of 

directed  energy  weapons,  "Navy  officials  admit  there's 
plenty  of  hard  work  ranging  from  basic  science  to  rigorous 
operational  evaluation  -  to  be  done  before  some  of  these 
systems  sail  aboard  a  warship.  Even  so,  it's  a  generation 
of  weapons  that  is  tantalizingly  close  to  becoming 
reality."  The  good  news  is  that  someone  is  giving  some 
thought  to  the  issue  of  evaluation,  meaning  that  the  notion 
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of  test  and  evaluation  is  not  lost  in  the  active  pursuit  of 
a  desirable  new  technology  and  corresponding  weapon  system 


which  will  certainly 

bend  the 

envelop  for 

testing  at 

our 

present  ranges. 

and 

using 

our  present 

targets . 

In 

addition,  during 

the 

month  of 

October  2003 

,  ITEA  will 

be 

conducting  a  symposium  on  understanding  direct  energy,  with 
implications  towards  T&E.  Perhaps  the  T&E  community  of 
practice  is  already  on  the  right  path. 

F.  AEGIS  T&E  -  LOOKING  FORWARD 

The  AEGIS  T&E  Process  has  always  been  intended  to  be  a 
universal  process,  with  applicability  to  both  government 
and  contractor  personnel  in  all  phases  of  the  acquisition 
cycle,  developmental,  operational,  or  combined.  Its  use 
implements  a  "Build  a  little  Test  a  little"  philosophy  and 
stresses  testing  before  expending  ordinance. 

Discipline  in  this  test  process  is  recognized  as  a 
contributor  to  cost  effective  system  acquisitions  that 
satisfy  the  Navy's  needs.  A  disciplined  and  well-structured 
test  program  reduces  the  risk  of  acquiring  an  ineffective 
system  and  provides  the  program  manager  with  timely 
information  required  to  make  prudent  decisions  during 
system  development. 

Testing  ranges  from  early  component  testing  at  the 
factory,  to  full  system  live  fire  performance 
demonstrations  in  a  simulated  real  world  environment. 
Regardless  of  the  type  of  test,  there  are  five  guiding 
principles  to  help  ensure  the  system  under  test  fulfills 
its  intended  purpose. 
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1.  Develop  meaningful  and  applicable  test 
objectives,  and  adhere  to  them  in  an  orderly,  repeatable, 
and  disciplined  manner. 

2.  Use  the  closed  loop  systems  engineering  approach, 
from  concept,  to  component,  to  subassembly,  to  subsystem, 
to  system,  to  whole  ship  test. 

3.  Test  as  early  as  possible  and  as  often  as 
affordable  to  find  and  correct  problems  before  they  become 
too  costly. 

4.  Involve  the  user,  developmental  tester  and 
operational  tester  in  the  initial  formation  of  the  systems 
engineering  council  to  develop  test  objectives  to  ensure 
continuous  and  timely  information  exchange  of  objectives 
and  test  results. 

5.  Take  the  time  to  ensure  all  parties  (developer, 
contractor,  and  government  operational  testers)  thoroughly 
understand  the  systems  mission  requirements  and  agree  on 
how  the  system  will  be  tested,  scored  and  evaluated. 

The  need  to  take  a  disciplined  approach  in  AEGIS  T&E 
has  been  demonstrated  many  times  in  the  past.  Risks  must 
be  understood  and  controlled.  Once  a  latent  deficiency 
manifests  itself,  it  is  no  longer  a  risk;  it  is  a  problem. 
The  AEGIS  Test  and  Evaluation  community  is  an  essential 
means  of  identifying,  understanding,  addressing  system 
issues  within  both  hardware  and  software. 

To  evolve  in  the  future,  AEGIS  T&E  will  complete  five 
objectives  for  each  improvement,  modification,  or  system 
mission  change: 
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1.  Verify  that  test  results  are  credible  and  support 
system  acquisition  milestones  for  decision-making. 
Incorporation  of  an  OA  software  implementation  system 
performance  should  be  the  same  if  not  better  then  the 
previous  legacy  system. 

2.  Provide  early  identification  of  AEGIS  performance 
and  supportability  deficiencies  for  resolution.  When 
limitations  are  discovered,  they  must  be  addressed  as  soon 
as  possible  to  support  further  tests  of  performance. 

3.  Identify  and  measure  performance  parameters  that 
are  critical  to  operational  effectiveness  and  suitability 
through  rigorous  analysis  and  evaluation  during  the 
evolution  of  system  requirements. 

4.  Provide  early  identification  and  timely  acquisition 
of  test  resources  and  assets  necessary  to  stress  the 
system.  T&E  assets  are  required  to  meet  the  approved  test 
objectives  and  provide  a  means  to  verify  specification 
compliance . 

5.  Execute  test  programs  that  consistently  apply  the 
closed  loop  systems  engineering  approach  to  T&E. 
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V.  CONCLUSION 


A.  RECOMMENDATIONS 

The  Navy  has  invested  billions  of  dollars  into  weapon 
systems  that  are  increasingly  complex  and  are  still 
evolving.  Changes  to  computer  program  architecture  and 
introduction  of  COTS  equipment  illustrates  the  Navy' s 
requirement  to  improve  existing  systems  and  hulls.  With 
this  evolution  T&E  culture  must  also  evolve  to  support 
future  weapon  systems  that  are  increasingly  complex  and 
agile.  It  is  not  possible  to  proceed  forward  doing  business 
the  way  it  has  been  done  in  the  past.  Change  in 
configuration  forces  the  evolution  of  T&E.  To  evolve  with 
the  systems  the  T&E  community  must  be  vigilant  in  the 
following  areas: 

Agility  -  To  be  adaptive  to  evolving  threats, 
increasingly  complex  weapon  systems,  and  more  and  more 
stressing  operator  training  needs.  The  T&E  Community  must 
evolve  with  the  systems  and  structure  the  evaluation 
performance  in  step  with  the  newly  imbedded  technology. 

Flexible  -  Being  able  to  address  whatever  new 
requirements  are  implied  with  the  improvements  or  upgrades 
to  the  systems. 

Meaningful  -  Bringing  to  the  event  the  regiment  and 
expertise  already  being  applied  to  legacy  systems, 
validated  test  objectives  and  measures  of  effectiveness, 
suitability  and  performance. 

Repeatable  -  The  T&E  community  must  be  able  to  sustain 
a  benchmark  for  regression  of  each  system.  Core 
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capabilities  may  be  improving,  but  each  ships  system  was 
designed  to  support  a  specific  mission.  No  matter  what  the 
change  in  computer  program  architecture  or  system  hardware 
improvement,  the  ships  system  must  fulfill  its  mission. 
Regression  testing  insures  compliance,  and  repeatability. 
The  T&E  community  must  be  capable  of  evaluation  of 
performance  to  verify  core  functionality  and  the  ability  of 
the  system  to  satisfy  its  mission. 

Innovation  -  The  T&E  community  must  find  solutions  for 
difficult  scenarios  blending  a  mix  of  live  and  modeled 
testing  to  gauge  system  performance,  and  to  also  provide  a 
value-added  operational  feel  for  the  warfighter.  OA  brings 
the  promise  of  greatly  enhanced  and  rapidly  upgradeable 
systems,  and  with  that  promise  comes  the  need  for  creative 
and  innovative  T&E  solutions. 

Expertise /Lessons -Learned  -  The  T&E  professional 
workforce,  who  is  the  backbone  for  conducting  modern-day 
T&E,  must  ensure  that  the  collective  knowledge  for  the 
business  of  test  and  evaluation  is  recorded,  and  passed  on 
to  the  next  generation  T&E  professionals.  The  AEGIS  T&E 
community  has  evolved  with  a  regiment  and  infrastructure 
based  on  lessons  learned  over  the  last  23  years  of  system 
test  and  certification.  As  the  AEGIS  program  puts  to  sea 
its  final  ships  and  the  AEGIS  Weapon  System  reaches  a  final 
configuration,  the  AEGIS  T&E  community  of  practice  must  be 
preserved  and  applied  to  future,  and  evolving  systems. 

Cost  Effective  -  T&E  must  provide  meaningful  and 
measurable  metrics,  which  demonstrate  conclusively,  the 
merit  to  T&E.  The  pitfalls  and  tradeoffs  from  inadequate 
testing  must  be  readily  available  to  help  tomorrow' s 
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decision  makers  to  defend  the  level  and  appropriateness  of 
T&E  in  the  future. 

Technology  Adopters  -  T&E  must  leverage  technology 
wherever  and  whenever  possible  as  a  workforce  multiplier, 
and  a  resource-saver.  Just  as  future  weapon  systems 
embrace  technology  to  provide  new  answers  to  difficult 
problems,  so  should  future  test  and  evaluation.  Up  front 
investments  in  technology  are  needed  to  ensure  this 
happens . 

Safety  -  Weapon  system  test  execution  must  remain 
safe.  Weapon  system  complexity  challenges  the  DOD' s 
ability  to  design  scenarios  to  adequately  understand  the 
performance-related  aspects  of  systems  undergoing  test. 
Regardless  of  testing  complexity,  safety  cannot  ever  be 
compromised.  Pressure  to  reduce  safety  standards  and 
practices  to  expedite  programs,  and  thereby  reduce  costs, 
must  be  resisted. 

Environmental  Compliance  -  Weapon  system  test 
execution  must  be  in  compliance  with  environmental  laws  and 
policies.  At-sea  testing  is  restrictive  and  difficult  to 
characterize  the  impact  to  the  environment.  New  future 
weapon  technologies  bring  the  challenge  of  additional 
review  for  environmental  compliance.  Increased  weapon 
system  complexity  further  challenges  our  understanding  and 
ability  to  estimate  impact  upon  the  environment.  The  time 
and  costs  associated  with  adequate  environmental  review  are 
prerequisite,  and  cannot  be  avoided. 

Test  Where  It  Makes  Sense  -  Current  sea-based  test 
ranges  typically  involve  test  areas  instrumented  for  live- 
fire  exercises  out  to  100  nautical  miles  offshore. 
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Increased  weapons  system  lethality,  range,  and  performance 
are  features  of  most  future  navy  weapon  systems.  To 
adequately  test  these  systems  at-sea  without  compromising 
safety  and  environmental  policies,  the  testing  is  being 
pushed  further  offshore,  away  from  traditional  land-centric 
test  range  infrastructure.  Because  offshore  waters  are 
being  encroached  more  and  more  by  commercial  and  private 
boat  traffic  and  air  traffic,  adequate  test  areas  free  from 
encroachment  must  be  pushed  further  away  from  traditional 
land-based  test  ranges.  Future  testing  will  need  to  be 
conducted  in  open-ocean  areas  using  both  remote  and 
autonomous  test  procedures/capabilities.  Major  development 
and  investment  in  unmanned  systems  operations  is  needed  to 
make  possible  open-ocean  testing.  Telemetry  (data 
collection)  systems  will  be  particularly  challenging.  New 
open-ocean  test  areas  might  need  200-400  nautical  miles  of 
instrumented  range.  This  requires  both  a  major  cultural 
change  as  well  as  increased  T&E  funding  resources. 

Affordable  -  T&E  processes  and  approaches  can  be  cost- 
effective,  yet  still  be  unaffordable.  The  costs  of  testing 
current  and  future  weapon  technologies  have  been  increasing 
for  more  than  two  decades.  The  ability  to  preserve  costly 
special  purpose  test  assets,  test  processes,  and  unique 
test  range  infrastructure  is  getting  more  difficult  and 
challenging.  T&E  complexity  is  directly  proportional  to 
weapon  system  complexity.  To  remain  affordable,  T&E 
Programs  are  now  expected  to  "get-it-right"  the  first  time 
to  keep  costs  affordable  and  manageable.  But,  T&E 
affordability  cannot  be  measured  in  the  test  infrastructure 
costs,  but  rather,  weapon  system  life-cycle  costs. 
Affordable  T&E  should  be  measured  by  the  degree  total 


weapon  system  life-cycle  costs  are  minimized  and  reduce 
associated  risks. 

B.  CONCLUSIONS 

Test  and  Evaluation  of  ships  systems  verify  that  the 
Navy  gets  what  it  paid  for,  and  the  systems  perform  the 
mission  they  were  designed  to  do.  Cost  effectiveness  and 
expedience  in  the  face  of  evolving  technology  are  the 
hallmarks  of  a  good  T&E  community.  The  lives  of  those  who 
operate  and  maintain  today' s  weapon  systems  depend  on  solid 
and  reliable  testing,  to  ensure  systems  do  what  they  were 
designed  to  do.  The  T&E  community  makes  it  possible. 

In  a  letter  to  the  editors  of  Scientific  American 
(Sawyer,  2003,  p.  20)  in  response  to  an  earlier  article 
entitled,  "Misguided  Missile  Shield",  the  writer  states,  "a 

demand  for  perfect  realism  in  testing  a  complex  weapon 

system  like  missile  defense  is  unrealistic.  More  testing 

in  necessary  -  more  tests,  however,  are  scheduled."  Indeed 
it  would  seem  that  testing  of  as  many  of  the  variables 

possible  is  prudent,  until  the  T&E  community  can  respond 
with  more  comprehensive  and  full-scale  tests  in 
environments  which  mirror  the  conditions  anticipated  where 
these  future  weapon  systems  will  be  operated  by  tomorrow' s 
warfighters . 


C.  SUGGESTED  TOPICS  FOR  FUTURE  RESEARCH 

This  research  provided  a  historical  account  of  several 
ongoing  and  emerging  Navy  T&E  programs  with  the  goal  being 
to  provide  a  series  of  attributes  T&E  must  exhibit  to 
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successfully  field  future  systems.  While  touching  on 
certain  indicators  and  spending  some  focus  on  AEGIS  Weapon 
System  development  and  open  architecture,  the  following 
topics  are  areas  that  should  be  considered  for  future 
research . 

•  Analyze  "lessons  learned"  from  evolutionary 
acquisition  to  show  how  programs  are  balancing  new 
capabilities  and  lifecycle  support  against  T&E 
abilities  and  needs.  T&E  must  evolve  and  transform  to 
provide  continuous  test  windows  inside  the  systems 
development,  as  well  as  being  as  operational  as 
possible . 

•  Assess  the  progress  of  AEGIS  Open  Architecture,  and 
show  mapping  against  Navy  Open  Architecture.  As 

standards  are  continually  being  developed  and  vetted 
out  in  the  technical  community,  the  real  success  lies 
in  bringing  actual  open  systems  direct  to  the 
warfighter.  Before  this  can  happen  however,  these 
standards  must  be  agreed  upon  and  current  and  emerging 
systems  will  have  to  adopt  them  unilaterally. 

•  Compare  and  contrast  the  decision  to  convert  the  AEGIS 
Fleet  into  an  open  systems  baseline,  versus  bringing 
the  AEGIS  Weapon  System  into  a  "caretaker"  status .  At 

present,  the  future  baseline  configuration  for  both 
AEGIS  Cruisers  and  AEGIS  Destroyers  is  still  a  matter 
of  great  debate,  and  very  much  dependent  on  future 
budgetary  decisions  and  an  unstable  political  horizon. 

•  Evaluate  the  challenges  of  providing  effective  joint 
and  allied  systems  T&E.  In  addition,  explore  the  need 
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for  consistent  interoperability  standards,  and  how 
open  systems  development  may  help  or  hinder  the 
interoperability  crisis  plaguing  many  major  in-service 
weapons  systems  today. 

•  Research  a  case  example  such  as  DD  (X)  as  a  "cradle  to 
grave"  open  architecture  program  currently  undergoing 
requirements  generation  and  definition  phase.  Explore 
the  techniques  for  building  an  open  computing 
environment  that  will  incorporate  test  and  evaluation 
inside  the  actual  tactical  code. 


The  1970  Blue  Ribbon  Defense  Panel,  also  know  as  the 
Fitzhugh  Commission,  took  a  very  serious  and  in-depth 
review  of  defense  acquisition  policies  and  procedures. 
Their  finding  led  to  sweeping  recommendations,  which 
changed  modern  acquisition  well  before  the  term 
"evolutionary  acquisition"  was  coined.  This  Commission 
also  made  profound  recommendations  concerning  T&E,  which 
actually  led  to  the  establishment  of  both  the  office 
overseeing  DT&E  as  well  as  OT&E.  In  the  thirty  plus  years 
since  the  Fitzhugh  Commission  made  their  recommendations, 
much  has  changed,  but  a  few  things  have  remained  the  same. 
The  T&E  community  must  never  forget  the  principal  reason 
for  testing  is  to  learn  and  to  gain  knowledge  and 
information  about  the  system  undergoing  design  and 
development.  No  matter  how  "open"  the  system  becomes,  this 
need  to  learn  remains,  and  testing  at  the  lowest  level,  to 
the  highest  (operational)  level  is  key.  But  testing  must 
be  done  by  experienced  professionals  using  proven  methods 
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and  given  adequate  recourses.  Is  the  current  T&E 
infrastructure  ready  to  handle  the  challenges  that  lie 
ahead? 

The  current  Director  of  Operational  Test  and 
Evaluation  recently  concluded  his  remarks  on  T&E  Role  in 
Experimentation  with  the  following: 

We,  the  T&E  community  -  in  both  industry  and 
government,  both  technical  and  operational 
testers  -  have  served  the  Department  very  well 
over  the  years . 

There  is  a  new  world  dawning  that  calls  for 
new  and  innovative  strategies  and  capabilities 
for  T&E.  I  am  confident  that,  together,  we  will 
rise  to  the  challenge  as  we  have  in  the  past  and 
ensure  that  our  soldiers,  sailors,  and  airmen  are 
equipped  with  the  best  equipment  our  nation  can 
provide  (Christie,  "Test  &  Evaluation's  Role  in 
Experimentation,"  2002.) 
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